At 4:16pm last Wednesday I got a short and to-the-point email from Nilay Patel at The Verge with only a link that started with the host “googlereader.blogspot.com”. The sudden spike in NewsBlur’s visitors immediately confirmed — Google was shutting down Reader.

Late night at the office

I had been preparing for a black swan event like this for the last four years since I began NewsBlur. With the deprecation of their social features a year ago I knew it was only a matter of time before Google stopped supporting Reader entirely. I did not expect it to come this soon.

As the Storify history of the Reader-o-calypse, NewsBlur suffered a number of hurdles with the onslaught of new subscribers.

A few of my challenges and solutions

I was able to handle the 1,500 users who were using the service everyday, but when 50,000 users hit an uncachable and resource intensive backend, unless you’ve done your homework and load tested the living crap out of your entire stack, there’s going to be trouble brewing. Here’s just a few of the immediate challenges I faced over the past four days:

• My hosting provider, Reliable Hosting Services, was neither reliable, able to host my increasing demands, or a service I could count rely on. I switched to Digital Ocean and immediately got to writing new Fabric scripts so I could deploy a new app/task server by issuing a single command and having it serve requests automatically within 10 minutes of bootstrapping.
• It didn’t take long to max out my Amazon Simple Email Service (SES) account’s quota of 10,000 emails a day. So a few hours into the melee I switched to Mailgun, which unfortunately resulted in emailing myself 250,000 error reports. If you tried to email me and couldn’t get through, it’s because 50,000 emails about lost database connections made their way ahead of you in line.
• Eventually, I was just plain blacklisted on SES for sending too many emails.
• Fortunately, when the PayPal fraud department called because of an unprecedented spike in payments, I was prepared.

Paypal’s fraud department just called, asked me what’s going on. Asked the rep from Omaha if she’s heard of Reader, and then a big Ohhh.

— NewsBlur (@NewsBlur) March 17, 2013

• HAProxy would Having HAProxy serve errors (site is down, maintenance, timeouts, etc) with a 200 OK status code instead of the proper 500 Exception status code because of a ridiculous undocumented requirement to include HTTP Headers at the top of the error template. When your webapp uses status codes to determine errors, you get extremely strange behavior when it loads utter crap into your DOM.
• The inevitable file descriptor limits on Linux means that for every database connection you make, you use up one of the 1,024 file descriptors that are allocated to your process by default. Changing these limits is not only non-trivial, but they don’t tend to stick. This is responsible for bringing down Mongo, PostgreSQL, and the real-time Node servers, all at different times of the night.
• The support queue is enormous and I’ve had to spend big chunks of my 16 hour days reassuring paying customers that eventually Stripe will forgive me and my unresponsive servers and will send the payment notification that is responsible for automatically upgrading their accounts to premium.

The sad extent of my St. Patrick’s Day

As a one-man-shop it has been humbling to receive the benefit of the doubt from many who have withheld their judgment despite the admittedly slow loadtimes and downtime NewsBlur experienced. Having the support of the amazing NewsBlur community is more than a guy could ask for. The tweets of encouragement, voting NewsBlur up on replacereader.com (If you haven’t yet, please tweet a vote for “#newsblur to #replacereader”), and the many positive comments and blog posts from people who have tried NewsBlur is great.

It has also been a dream come true to receive accolades from the many who are trying NewsBlur for the first time and loving it. Since the announcement, NewsBlur has welcomed 5,000 new premium subscribers and 60,000 new users (from 50,000 users originally).

NewsBlur users are intelligent, kind, and good looking!

The next three months

Over the next three months I’ll be working on:

• Scaling, scaling, scaling
• Launching the redesign (which you can preview)
• Listening to all of you

For those of you who are still trying to decide where to go now that you’re a Reader refugee let me tell you a few of the unique things NewsBlur has to offer:

1. Radical transparency. NewsBlur is totally open source and will remain that way.
2. It still feels like RSS, just with a few more bells and whistles. NewsBlur provides actual list of posts, as opposed to the more curated magazine format of some of the other popular replacements. This clean interface makes it easy to see the stories you want. One innovation however is the four different view options you have. NewsBlur can show you the original site, feed, text or story view.
3. It has training. NewsBlur hides stories you don’t want to read based on tags, keywords, authors, etc. It also highlights stories you want to read, based on the same criteria. This allows you to find the stories you care about, not just the stories that the hive Hive cares about. And best of all, NewsBlur will show you why stories are either highlighted or hidden by showing the criteria in green or red.
4. NewsBlur has rebuilt the social community that Google had stripped out of Reader. Users can share stories through their Blurblog and discover new content by following friends’ Blurblogs. The People Have Spoken is the blurblog of popular stories.
5. Because NewsBlur is entirely open-source, if you don’t want to pay you can host your own server. Instructions are on GitHub, where you can also find the source code for the NewsBlur iPhone + iPad app and Android app.
6. Most importantly, NewsBlur is not entirely a free app. The immediate benefits of revenue have been very clear over the past few days. Not only are NewsBlur’s interests are aligned with its users, but as more users join NewsBlur, it makes more revenue that can be used to directly support the new users. Not convinced that paid is better than free? Read Pinboard’s Maciej Ceglowski’s essay Don’t Be a Free User.

Shiloh during better times. Your premium subscription goes to both server costs and feeding her

With NewsBlur’s native iOS app and Android app, you can read your news and share it with your friends anywhere. And with the coming improvements over the next three months, you bet NewsBlur will be the #1 choice for Google Reader refugees. survivors.

Join NewsBlur for $24/year and discover what RSS should have been.

Luther’s consolation when things are tough, especially in the church:

When we are plunged into disasters and troubles and covered by darkness, things on account of which we cannot be sure that we are the church or pleasing to God, let us learn to take hold of the Word and let sink and fall what falls, and let us not be moved by the fall and defection of others. But let us reflect that we are in a dark place, with the lamp of the Word shining before us. “For he who believes and is baptized will be saved” ( Mark 16:16). For that light is the only one which the sun and human reason do not see; but it shines in the heart. Besides this Word we should know nothing and see nothing. For if this Word alone is shining, there is no danger, and the hour will come when we shall come forth and say with joy: “I have seen the Lord face to face, and my soul has been saved.”

AE 6

Repent!

Man, yesterday was great. I went out early to that supermarket that bakes fresh 7 days-a-week and picked up Pączki for the family for breakfast. Friends introduced us to this Polish Tuesday-before-Ash Wednesday tradition and we have enjoyed Pączki Day for over 30 years. In a similar vein German Americans have traditionally celebrated with a traditional doughnut donut called Faschnacht. If you background leans a little more to being English, yesterday would have been Pancake Day.

Whatever the cultural variation, the day had a purpose, eating rich foods on the Tuesday-before-Ash Wednesday Tuesday was a way to empty the pantry of lard, sugar, fat, and butter, which were traditionally fasted from during Lent.

And then an let’s not forget those who celebrate Fat Tuesday with a party or carnival—for two weeks St. Louis French quarter, Soulard, has been host to the second largest Mardi Gras in the country—and Mardi Gras is simply French for ‘Fat Tuesday.’ Just about wherever Lent is celebrated, Fat Tuesday traditions have grown up.

Like many traditions, while the ‘fun’ and ‘traditional’ aspects endure and are celebrated far and wide, the reason for the tradition is largely forgotten or set aside as no longer relevant. Why would Christian households commit the day before Ash Wednesday to emptying the pantry of butter, fat, and sugar? The answer is not in the donuts or the pancakes, or the rich Mardi Gras foods—as much as we like them. With the pantry cleaned out, the Christian family is prepared for the traditional fasting that is associated with Lent.

A few of the liturgical traditions still remember or use the old term Shrove Tuesday. ‘Shrove’ means to have been shriven before Lent, an old English way of saying to [be prepared] to obtain absolution for one’s sins by way of Confession. Here then is the other traditional aspect of Lent: that is confession so that one may obtain absolution for sin. Lent prepares us to approach the Cross on Good Friday, approach shriven, approach having contemplated our sins, not just to have made a list of them, but to have pondered anew the great need we have for a Savior from our sin; to contemplate the gravity of our sin—sin so deep, so dark, that it required nothing less than that the Son of God would die that we could be reconciled to God.

Ash Wednesday is set aside yearly as a day of repentance, a day to contemplate your sins and to be sorry for your sins. Not matter your Christian tradition, it is not too late to be shriven—and it doesn’t require you to go to IHOP or Drive-In Donuts. Repent! Hold up your life, your thoughts, your pet sins before the mirror of God’s Ten Commandments and see how far we are from His holy will.

But to recognize sin is not enough, to make a list of all our sins, even if we could, is not enough. Repentance, true repentance, requires that you turn away from your sin. To turn away from excess during Lent just to pick it up again in spades after Easter is no true Lenten discipline. In the same way, to confess your sin, but not turn aside from it, to not change your life, your actions, your sinful behavior, but to return to them—that is not true repentance. Empty ‘sorrys’ come so easily off our lips. Like many who celebrate Fat Tuesday with no intention of being shriven, we throw ‘sorrys’ around with no understanding or intention to do anything really different. Your “I’m sorry,” no matter how heartfelt, is insufficient if it is not joined with the change needed to not commit that sin again.

God despaired of our sin. Yet, instead of acting in righteous judgment against us, He sent His Son to die for our sin, so that we might live with Him forever. The weight of your sin–my sin, the heavy yoke of it, crushed the Lord on the cross, so that we would not have to be crushed into the grave forever. He drank to the dregs the bitter cup of God’s wrath against our sin, so that in true faith we receive the sweet cup of blessing in Christ.

While we turn our attention to confessing our sin on this Ash Wednesday, all the more let us turn our lives to lives of repentance-lives where we continually hold ourselves accountable to God and to each other, lives where we sorrow over sin AND seek the God-given strength to turn from our sins. This is the work the Holy Spirit does in us, the daily drowning of the old sinful man and the bringing forth of the New Man—this is the daily remembrance of our Baptism that strengthens us.

Confession has two parts, first, to confess our sins, and second to receive absolution. Absolution, the sweet word of God that announces that your sins are forgiven. Absolution, the sweet Gospel that on account of Christ, God sets aside the punishment of your sin.

It’s Ash Wednesday. Repent! Then hear the Word of forgiveness with the assurance that your sins are truly forgiven by God in heaven.

Amen.

Almighty and everlasting God, You despise nothing You have made and forgive the sins of all who are penitent. Create in us new and contrite hearts that lamenting our sins and acknowledging our wretchedness we may receive from You full pardon and forgiveness; through Jesus Christ, Your Son, our Lord, who lives and reigns with You and the Holy Spirit, one God, now and forever. (L22)

Previous Post on Ash Wednesday