One of my predictions failed :-(

In 2012, long before I started this blog but a few years into my unofficial side career as a futurist, I predicted that 1 terabyte (TB) thumb drives would cost no more than $20 apiece by the end of 2019. I was wrong.

I made that prediction in the form of a Facebook Note, which I’ve copied and pasted to the end of this blog entry (see far below). At the time, I used Kryder’s Law (the observation, first made in 2005, that hard drive density doubles every 13 months) and did some back-of-the-envelope calculations to extrapolate price trends in solid state memory, leading to my 2019 date. Prices didn’t come down as fast as I predicted, and today $20 will at best get you a 256 GB thumb drive. I saw that offer during a Black Friday sale, when retailers usually offer the lowest prices of the year, and in this case, of 2019. This means I fell two price-doublings short.

(Quick aside: Other solid-state memory deals I saw on Black Friday 2019:

• 1 TB Seagate Desktop SSHD 7200 RPM internal hard drive – $35 (Newegg.com)
• 1 TB Western Digital EasyStore external hard drive – $40 (Best Buy)
• 128 GB SanDisk MicroSD card – $14 (Walmart))

I mistakenly assumed that Kryder’s Law applied to the sorts of solid-state computer memory chips found in thumb drives. In fact, the Law only applies to the older type of rotating, magnetic hard disk memory drives, meaning I had even less of a foundation for my trend calculations. If there is no fundamental force of science, technology, industry, or nature undergirding an observed phenomenon, then there’s no reason to expect the phenomenon to continue. I had correctly observed that thumb drives were getting cheaper year over year, but assumed without basis that the improvement would continue at that same rate until 2019.

However, even if Kryder’s Law had applied to solid-state memory, it wouldn’t have saved my 2012 prediction since, in the years after, the Law stopped holding true. The graph below shows the average cost-per-byte of HDD space from 1990 – 2005. Note the graph has a logarithmic scale, and the blue price points neatly form into a non-horizontal line, indicating an exponential trend. You can understand why Mark Kryder looked at the data in 2005 and created his eponymous Law of exponentially improving price performance (specifically, with a doubling every 13 months). Extrapolating the Law into the future, as indicated by the red line, a 1GB HDD should have only cost one cent by the end of 2019. This means a 1TB HDD should have cost $10.

2019 is now nearly over, and the cheapest, newly manufactured 1TB HDD I found in my research cost $35. The difference is because, after 2005, HDD prices stopped decreasing at the rates Kryder had observed.

The trend’s downward slope flattens a little from 2005-2011, and then flattens A LOT from 2012 onward. Muddling the data is the fact that massive floods hit Thailand in 2011, which disabled several important computer chip factories, reducing global HDD supplies and spiking their prices. However, after the last of those factories was restarted in 2013, the cost-per-byte trend didn’t return to its pre-2011 downward slope. The slope since 2013 has been much shallower.

The sharp slowdown in progress is thanks to the current HDD technological paradigm, called “perpendicular magnetic recording” (PMR), reaching the limits of what it can achieve. The next technological paradigm, called “heat-assisted magnetic recording” (HAMR), has been delayed by several years because various engineering and reliability problems have proven harder to solve than expected. In fairness, Mark Kryder couldn’t have foreseen this in 2005.

So yup, I was wrong. I own up to it, understand the reasons for my mistake, and won’t repeat it. So let me do a new prediction, this time based on more relevant data, and more cautiously couched. Here are historical price data for flash memory:

Eyeballing the scatterplot, the rate of price-performance improvement slowed down a lot around 2010. I don’t know what happened then, but there’s enough of a disconnect for me to say that the trend could best be represented with two, downward-sloping straight LSRLs (least-square regression lines):

The horizontal purple line represents the $20 mark. The yellow line depicts the old cost trend, and had it continued, a the cost of a 1TB flash drive would have dropped to $20 in 2014. However, for reasons unknown, we’re now operating under the shallower red line, and it doesn’t intersect with the purple line until the middle of 2022, which suggests that the 1TB/$20 milestone will happen by the end of that year.

I believe that the red line trend will persist until at least 2022 because it is being largely driven by advances in 3D NAND “chip stacking” techniques, and the technological paradigm doesn’t seem like it will reach its limits in the next three years. Thumb drives, like the one made by “Patriot Memory” I showed a picture of, have about 64 flat memory chips, stacked vertically like a pack of cards. Adding an extra layer increases the device’s overall memory storage capacity, while raising the cost of manufacture by a disproportionately small amount. This year, semiconductor companies started mass producing flash drives with 128 layers of chips, and it shouldn’t be long before they are incorporated into common thumb drives, resulting in a near-doubling of price-performance. It’s unclear how far the “layer stacking” method can go before it hits a technical/cost wall (at some point, the marginal downsides of adding a new chip layer exceed the benefits thanks to longer manufacturing times, higher costs, and unacceptably high defect rates), but for what it’s worth, experiments are now underway to make 176 layer chips, and some semiconductor engineers believe the ultimate practical limit is somewhere in the hundreds of chip layers.

Even if the practical limit to the height of the chip stacks arrives before the end of 2023, another doubling of 3D NAND price-performance could be had by finding ways to shrink the sizes of the individual cells that store bits of data on each chip. Shrinking cell sizes from the current 40nm to an entirely doable 30nm would almost double the price-performance. (Older, single-layer flash chips have 15nm cells, which are much harder to make than 30nm cells.)

In summary, I think the current rate of price-performance improvement for thumb drives will continue until a 1TB thumb drive costs only $20. They will probably be that cheap by the end of 2022, but because I’m cautious, I predict the milestone will be reached by the end of 2023.

Links:

1. Mark Kryder bio – https://en.wikipedia.org/wiki/Mark_Kryder
2. 2014 analysis showing that Kryder’s Law had failed – https://www.semanticscholar.org/paper/An-Economic-Perspective-of-Disk-vs.-Flash-Media-in-Gupta-Wildani/a60e27abf3eda07dc5bf383b08f8027f9277dd93
3. 2017 article about 3D NAND and its technical challenges – https://www.theregister.co.uk/2017/08/03/flash_layer_cell_shrink_tech/
4. 2019 announcement of 128 layer 3D NAND – https://www.anandtech.com/show/14589/sk-hynix-128-layer-4d-nand
5. More on how to improve 3D NAND – https://www.theregister.co.uk/2018/01/15/qlc_3d_nand_error_correction/
6. Source for all of my HDD price data – https://jcmit.net/diskprice.htm
7. Source for most of my flash memory price data – https://jcmit.net/flashprice.htm

===========================================================

(My original prediction, published on November 23, 2012)

2019: Your life on a cheap thumbdrive

In late 2005, I bought my first thumbdrive. It cost $20 and only had 1 GB.

That means that, in seven years, the cost-performance of flash memory has undergone about 5.5 doublings.

If the trend continues, in another seven years (2019), $20 will buy you a 1 terabyte (TB) thumbdrive. A terabyte is 1,000 gigabytes.

So what? Why care? Think about how big 1 TB is:

You could fit more than 300,000 high-res digital JPEG photos from a good D-SLR, or 250,000 full-length MP3 songs onto 1 TB. As massive as your digital photo and music collections are, they don’t come anywhere close to maxing out 1 TB. Go check your files now if you don’t believe me. You’d be lucky to break the 100 GB mark.

Let’s go a step farther and assume that you scanned all of your old film photos into your computer as well. Even doing color scans at 600 DPI (which is very hi-res), each individual photo will be at most 6 MB in size. Even if you had 10,000 old photos from the pre-digital days (which you almost certainly don’t), it would all take up only 60 GB.

Now, go a step farther: Tally up the filesize of all your email accounts, all your saved Word documents and misc personal files on your PC hard drive, your Facebook account, and any other worthwhile personal digital data you have. Add it to all the rest, and I’ll bet you’re still not close to the 1 TB mark.

Take another step and also estimate how big your stock of important personal papers (i.e. – Social Security card, driver’s license and other forms of ID, old report cards, old handwritten letters, drawings, diaries, financial statements, medical records, etc.) would be if you scanned them all. Assume each page is 8.5″ x 11″, color scanned, and done at a 300 DPI resolution (which is more than adequate for written documents). Let’s be generous and assume that each resulting JPEG file is 1 MB. Even 10,000 pages of scanned stuff only takes up 10 GB.

Throw in all your scanned VHS home movies, and any other barely relevant archives of your life, and you’re probably still not close to the 1 TB mark.

So, by 2019, you’ll be able to fit almost all the documents that describe who you are, what you like, and what you did onto a $20 device that is smaller than your pinkie finger. And as needed you could copy all that data onto other cheap backup devices in the space of a few minutes. There’s something truly surreal about that, and it really drives home how much our technology is surging past the familiar human pace of thinking, living, and generating meaningful content.

The only way you could easily break the 1 TB barrier for personally relevant computer files is if you started constantly recording your life with cameras in hi-def 1080p. If you set up such cameras throughout your house, in your car, and maybe on your person in order to permanently record every boring second of your existence, then you would blow past 1 TB pretty fast.

Such a practice is called “lifelogging,” and I think it will become common in the 2020’s as hard disk prices drop orders of magnitude lower than the $20 per TB example discussed in this Note, and as hi-res cameras become tiny and dirt cheap. If we’re wearing augmented reality glasses by then, they will be embedded with 1080p cameras and microphones, and you could easily set it to constantly record everything and upload it onto some central hard drive where you keep all your files. AI by that point should be good enough to actually understand much of what’s going on in your recordings, so you could verbally ask your Google Glasses something like: “Hey, what was the name of that guy with the red hair and leather jacket that I met last month at that dinner?” and it would be able to scan through your past recordings and find the answer for you.

But I’ve gotten off-track…just know that we are entering an age in which everything will be recorded and stored digitally forever. Near-perfect records of everything that happened, everything that was written, and everything that was said will exist by 2030. With instantaneous access to their lifelogs, no one would ever forget anything. The fuzziness and subjectivity of human memory would be superseded by clear, objective recordings. And with cameras all over the place and being constantly carried around by random people, it will be very hard to escape detection and to live anonymously.

Comment added later: I forgot something: By 2019, you will also be able to get you personal genome sequenced for less than $1,000 and store it digitally in your $20 thumbdrive. Your DNA should take up at most 2 GB of storage space if compressed.