Notes on scale + quality

A renewed interest in human curation, a slow move away from big social, a clearer understanding of platform incentives, the economic feasibility of subscription businesses, and builders driven by a new set of values – combined, it feels like a good time for Internet businesses that are both human-scale and profitable.

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more

 You walk down your high street. What do you prefer to see there? The economist will say: Walmart, Best Buy, the Gap. Scale economies — cheaper prices — better for “consumers”! But the human being will say: an independent cafe, a good bookshop, a boutique clothing store. Why? Because they offer many things that mega scale organizations don’t.

Like a band that is accused of selling out, there is an inevitable sense that a certain sharpness of flavor, of choice, has seeped out as more and more people join up and as a service loosens up and accommodates more more use cases. Another form of diseconomy of scale is behind the flight to Snapchat among the young, as outlined earlier.