1
00:00:01,000 --> 00:00:08,383
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,383 --> 00:00:52,316
[Hal Turing] Today we're looking at ANNS-AMP: Accelerating Approximate Nearest Neighbor Search via Adaptive Mixed-Precision Computing. The first author is Mingkai Chen, with Cheng Liu, Shengwen Liang, Lei Zhang, Xiaowei Li and Huawei Li. They're at the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. It's an arXiv paper from June 2026. The question is whether a small runtime predictor can pick just enough numeric precision for each cluster of a vector index, and whether a bit-serial accelerator can turn that into real speed and energy savings.

3
00:00:52,316 --> 00:01:14,746
[Dr. Ada Shannon] The stake is that vector search sits underneath retrieval for LLMs and recommendation systems, and the paper says about 99.9% of its distance arithmetic is spent on vectors that are not neighbors. If you can spend fewer bits on that waste, retrieval gets cheaper. The catch is whether the savings come from the idea or from the custom silicon around it. We'll get there.

4
00:01:14,746 --> 00:01:18,322
[Hal Turing] Start with the basics. What is approximate nearest neighbor search?

5
00:01:18,322 --> 00:01:37,362
[Dr. Ada Shannon] You have a huge pile of embedding vectors and a query vector. You want the few vectors closest to it. Exact search compares against everything, which is too slow, so you accept a small chance of missing a true neighbor in exchange for speed. The neighbor set is tiny next to the corpus, which is why almost all the distance work is wasted.

6
00:01:37,362 --> 00:01:39,731
[Hal Turing] So how does the index avoid touching everything?

7
00:01:39,731 --> 00:02:09,870
[Dr. Ada Shannon] This paper uses the IVF-PQ style, as in Faiss. IVF means vectors are grouped into clusters, and a query probes only the nearest few, a setting called nprobe. PQ, product quantization, from Jégou, Douze and Schmid in 2011, compresses each vector. You split it into M pieces, and each piece becomes a one-byte code pointing into a small codebook. That's similar in spirit to weight quantization in networks, except there's no gradient. It's just k-means.

8
00:02:09,870 --> 00:02:11,263
[Hal Turing] Walk me through a query.

9
00:02:11,263 --> 00:02:49,158
[Dr. Ada Shannon] Five stages. Say the query is a 128-dimensional vector. First, cluster locating: compare the query to every cluster centroid and keep the closest few. Second, residual calculation: subtract each chosen centroid from the query. Third, lookup-table construction: for each of the M pieces, compute the distance from the query piece to every codebook entry. That table is the key object. Fourth, distance calculation: a database vector's distance is just the sum of M table lookups, with no arithmetic on the original vector. Fifth, top-k selection.

10
00:02:49,158 --> 00:02:50,366
[Hal Turing] And which stages hurt?

11
00:02:50,366 --> 00:03:03,926
[Dr. Ada Shannon] Cluster locating and table construction. Those are the two stages that do real distance arithmetic. The paper reports that over 99.9% of that work goes to non-neighbors in both. So that's where the redundancy is.

12
00:03:03,926 --> 00:03:07,781
[Hal Turing] People already try to skip wasted work, though. Why isn't that enough?

13
00:03:07,781 --> 00:03:43,075
[Dr. Ada Shannon] Three families, each with a gap. Index tuning, picking cluster counts and PQ settings offline, is static. It doesn't adapt to the query. Early termination stops a distance computation once the partial result is clearly worse than the current top-k threshold. That needs a top-k list, which is loose early in a search. And table construction doesn't rank candidates, it builds a table, so there's no threshold to compare against. The paper counts essentially nothing removed from that stage by those methods. JUNO, which uses ray-tracing hardware for grid-based pruning, runs into trouble in high dimensions.

14
00:03:43,075 --> 00:03:46,001
[Hal Turing] So the proposal is to keep computing, just coarsely.

15
00:03:46,001 --> 00:04:21,156
[Dr. Ada Shannon] Right. Adaptive mixed precision. Classic mixed precision, like Micikevicius's 2018 FP16 training, is static: a human or a rule picks the types. HAQ and HAWQ pick bit-widths per layer, but still offline. Here the bit-width is chosen per cluster, per query, at runtime. The reasoning is that ranking doesn't need exact distances. A cluster far from the query can be rejected with very few bits. A cluster near the cutoff needs more. The data starts as 8-bit integers, so low precision here means below 8.

16
00:04:21,156 --> 00:04:25,335
[Hal Turing] But a GPU doesn't have a 5-bit multiplier. How do you cash that in?

17
00:04:25,335 --> 00:04:57,658
[Dr. Ada Shannon] That's the hardware half. Bit-serial arithmetic handles one bit of each operand per cycle, so time scales with bit-width. Halve the bits, double the throughput, with no idle lanes. The Stripes paper, by Judd and colleagues in 2016, argued that per unit area, bit-serial units can match the throughput of multi-bit ones. Pair that with a bit-plane memory layout, where you store the top bit of many values together, then the next, and a low-precision cluster fetches fewer bytes too. That matters because search is usually bandwidth-bound.

18
00:04:57,658 --> 00:05:08,850
[Hal Turing] Hold on. Stripes, Loom, Bit Fusion all made this pitch for neural networks, didn't they? And mainstream hardware went to fixed INT8, FP8, FP4 instead.

19
00:05:08,850 --> 00:05:33,649
[Dr. Ada Shannon] They did. Per-layer flexible precision mostly got bypassed by simply adding narrower fixed formats. So the fair question is what's different for retrieval. One candidate: the needed precision depends on the data and query, not on a layer you can fix in advance, and CPUs and GPUs have little sub-8-bit support to fall back on. Whether that's enough to break the pattern is something to hold up against the results.

20
00:05:33,649 --> 00:05:41,358
[Hal Turing] And the other thing to hold up is the one you flagged. This is a custom near-memory chip, so how much of the win belongs to the precision idea?

21
00:05:41,358 --> 00:06:00,816
[Dr. Ada Shannon] Keep both questions in mind: how do the predictor and hardware work, and what do the comparisons isolate? Start with the predictor, because the hardware only pays off if the predictor finds real slack. Its job is to say, for each cluster and each query, how many bits are enough to tell true neighbors from non-neighbors. For that the paper first has to divide the space sensibly.

22
00:06:00,816 --> 00:06:04,624
[Hal Turing] Why not just cut the space into a grid? That's what JUNO does, right?

23
00:06:04,624 --> 00:06:31,048
[Dr. Ada Shannon] Yes, and grids scale badly. In D dimensions a cell touches on the order of D neighbor cells, and a query near a border has to look in those too. Clusters avoid that, but the paper's Equation 1 shows the catch: clusters overlap. The chance that a candidate in your own cluster beats one in another cluster rises when the gap between the two centers is more than about four radii. In 128 dimensions, clusters sit too close for that to hold.

24
00:06:31,048 --> 00:06:32,209
[Hal Turing] So shrink the radius.

25
00:06:32,209 --> 00:06:56,730
[Dr. Ada Shannon] Right, and the trick is to split the dimensions into slices before clustering. Clusters that overlap in the full space often separate in a lower-dimensional projection, because the radius shrinks. The ablation is blunt. With no dimension splitting, zero sub-spaces qualify for low precision. At 16 slices, 87.5% do. At 32 slices the share falls again.

26
00:06:56,730 --> 00:06:58,912
[Hal Turing] And the predictor itself? What does it look at?

27
00:06:58,912 --> 00:07:25,290
[Dr. Ada Shannon] Five things: the distance from the query to the cluster, the radius of that cluster, the radius of the nearest cluster, and the two cluster sizes. Labels come offline from ground truth. If a cluster holds true neighbors, the label reflects the gap between those neighbors and the nearest non-neighbors. If it holds none, the label reflects the gap between its closest vector and the farthest true neighbor. A bigger gap means fewer bits can separate them.

28
00:07:25,290 --> 00:07:28,216
[Hal Turing] Why a support vector regression, and not a small neural net?

29
00:07:28,216 --> 00:07:55,430
[Dr. Ada Shannon] The paper doesn't compare against other model families, so I can't say. What it does show is that the kernel matters. A linear kernel found no low-precision opportunity at all, so the relationship is non-linear. They use an RBF kernel, evaluated through a lookup table to avoid exponentials. Training cost grows with the square of the sample count, so they use at most 1280 samples, with separate hyperparameters for cluster locating and table construction.

30
00:07:55,430 --> 00:07:59,470
[Hal Turing] Now the hardware. Prediction sounds like extra work that eats the savings.

31
00:07:59,470 --> 00:08:21,018
[Dr. Ada Shannon] That's what the reuse is for. The main compute array has bit-serial subtract, multiply and add units. A separate prediction module finds the nearest cluster and the query-to-cluster distances on that same array and the same priority-queue hardware, so there's no second array. Adder trees sum the partial distances from each slice, and priority queues do the top-k.

32
00:08:21,018 --> 00:08:23,851
[Hal Turing] Fewer bits computed doesn't mean fewer bits fetched, though.

33
00:08:23,851 --> 00:08:43,959
[Dr. Ada Shannon] Exactly, and without a fix the chip becomes memory bound. So the layout stores the top bit of many values together, then the next bit, and so on. A cluster at six bits reads six contiguous planes and skips the rest. In the cluster-locating stage that gives more than 1.18 times fewer memory accesses than an ordinary layout.

34
00:08:43,959 --> 00:08:46,049
[Hal Turing] What about units finishing at different times?

35
00:08:46,049 --> 00:09:07,829
[Dr. Ada Shannon] An idle group signals a busy neighbor to hand off work, guided by a simple estimate from cluster size, dimension and precision. It barely matters at default settings: about 3% more throughput in table construction, negligible in cluster locating. With a looser accuracy target it reaches about 1.15 times, because the bit-widths vary more.

36
00:09:07,829 --> 00:09:09,873
[Hal Turing] Okay, the setup, then the results.

37
00:09:09,873 --> 00:09:42,659
[Dr. Ada Shannon] It's a simulated 14-nanometer design: 7.5 square millimeters, 11.5 watts, 1 gigahertz, on stacked memory with 1600 gigabytes per second. Data is SIFT100M and DEEP100M, 100 million vectors each, 10,000 queries, recall at 10 of at least 0.8. Baselines are Faiss on a Xeon Gold 5218, Faiss on an A100, and ANNAx12, the authors' own simulation of an earlier accelerator, with their chip throttled to 800 gigabytes per second to match.

38
00:09:42,659 --> 00:09:43,727
[Hal Turing] And the headline?

39
00:09:43,727 --> 00:10:06,947
[Dr. Ada Shannon] About 164 times faster than the CPU, 10.6 times faster than the GPU, and 2.06 times faster than ANNAx12. Energy is roughly 1100 times, 39 times and 6.7 times lower. Read the first two against the last. The CPU gap is about 80 times the ANNAx12 gap, and the GPU gap about five times.

40
00:10:06,947 --> 00:10:10,430
[Hal Turing] So the 2x row is the one that says something about the precision idea.

41
00:10:10,430 --> 00:10:17,629
[Dr. Ada Shannon] It's the nearest thing to it. It still bundles in the layout and scheduler, so it isn't a clean precision test. I'll come back to that.

42
00:10:17,629 --> 00:10:20,183
[Hal Turing] What did the predictor actually buy, in its own terms?

43
00:10:20,183 --> 00:10:41,499
[Dr. Ada Shannon] Of the distance work in building lookup tables, over 93% runs below eight bits, even though the data starts as eight-bit. In cluster locating it's 75% to 87%. Overall accuracy loss stays under 2.7% in the main sweep, though some sub-space settings reach 3.98%. The paper doesn't define whether that loss is relative or in points.

44
00:10:41,499 --> 00:10:43,403
[Hal Turing] Why does cluster locating get less?

45
00:10:43,403 --> 00:10:52,691
[Dr. Ada Shannon] A wrong guess there can make the search miss a whole cluster, so the predictor has to be more conservative. Table construction mistakes are milder, so the chip takes fewer bits.

46
00:10:52,691 --> 00:10:55,059
[Hal Turing] Last piece. Does it scale with memory bandwidth?

47
00:10:55,059 --> 00:11:05,137
[Dr. Ada Shannon] Throughput flattens near 2000 gigabytes per second at default settings. Beyond that, they'd need more compute. So the design sits close to the memory ceiling, which is where the layout matters.

48
00:11:05,137 --> 00:11:13,264
[Hal Turing] Let's put the central question to the evidence. How much of the headline belongs to adaptive precision, and how much to being a custom near-memory chip?

49
00:11:13,264 --> 00:11:36,437
[Dr. Ada Shannon] The paper doesn't isolate it. The missing experiment is the same accelerator, the same bit-serial array and the same layout, run at fixed 8 bits with no predictor. Without it, the precision contribution is unknown. The 164 times over the CPU and the roughly ten times over the A100 mostly measure an ASIC with 1600 gigabytes per second of stacked memory against general-purpose processors.

50
00:11:36,437 --> 00:11:37,830
[Hal Turing] And the energy comparison?

51
00:11:37,830 --> 00:11:59,007
[Dr. Ada Shannon] It compares a simulated 11.45 watt chip against measured whole-processor energy, from RAPL on the CPU and nsys on the GPU. Those are different accounting boundaries, so the 1100 times and 39 times figures shouldn't be read as the effect of precision. No error bars or run-to-run variance are reported either.

52
00:11:59,007 --> 00:12:04,162
[Hal Turing] So ANNAx12 is the row that matters. About twice the speed, 6.7 times the energy.

53
00:12:04,162 --> 00:12:27,103
[Dr. Ada Shannon] It's the closest thing to an isolated contribution, but it doesn't isolate precision. That gain also includes the layout and the scheduler. ANNA's authors didn't release their code, so this team built its own simulator and checked it against typical results in the ANNA paper. They report no calibration error. They say the process technology, frequency and bandwidth match, but not whether the buffers and the HBM model match.

54
00:12:27,103 --> 00:12:30,447
[Hal Turing] So a modest modelling error in the baseline could eat a 2x gap.

55
00:12:30,447 --> 00:12:34,859
[Dr. Ada Shannon] It could. I'm not saying it does. I'm saying the paper gives us no way to rule it out.

56
00:12:34,859 --> 00:12:40,571
[Hal Turing] What about ANSMET, the bit-plane early-termination work from Yiwei Li and Mingyu Gao in 2025?

57
00:12:40,571 --> 00:13:06,484
[Dr. Ada Shannon] That's the most relevant prior work, because it also fetches bit-planes. But the comparison uses numbers estimated from ANSMET's published results, on a graph index, against this cluster index. The paper's own explanation is that graph search makes many random accesses. So the table mostly compares index families. It doesn't test whether prediction beats threshold-based pruning. A same-index, same-simulator run would.

58
00:13:06,484 --> 00:13:14,658
[Hal Turing] Back to the precedent from earlier: Stripes, Loom, Bit Fusion. Why did per-layer precision lose to fixed INT8?

59
00:13:14,658 --> 00:13:44,843
[Dr. Ada Shannon] Fixed narrow datatypes were simpler, and mainstream hardware adopted them. What might differ here is that the precision depends on the data. It's chosen per cluster and per query, with no top-k dependence, and the work is lookups rather than multiply-accumulates. What could repeat history is that the data already starts as uint8, so low precision only means below eight bits. Conservative widths dominate at recall 0.8, and the scheduler barely helps at default settings.

60
00:13:44,843 --> 00:13:47,165
[Hal Turing] The 2.7 percent loss also needs unpacking.

61
00:13:47,165 --> 00:14:07,367
[Dr. Ada Shannon] It does. Some settings reach 3.98 percent, and the text I have doesn't say whether that's a relative drop or points of recall. The target is recall@10 of 0.8, which is loose, and nothing is shown at 0.95 or at recall@100. The hardware also isn't fabricated. It's synthesized and simulated.

62
00:14:07,367 --> 00:14:08,481
[Hal Turing] And the workloads?

63
00:14:08,481 --> 00:14:34,999
[Dr. Ada Shannon] Two image-descriptor sets, SIFT100M and DEEP100M. The paper motivates itself with LLM retrieval and recommendation, yet it tests no modern text embeddings, no inner-product metric and no end-to-end task metric. The SVR was trained on at most 1280 samples and retuned for each setting, and the slice count matters a lot, so we can't tell how well it transfers to other data.

64
00:14:34,999 --> 00:14:39,921
[Hal Turing] There's also the software side. Faiss defaults aren't the strongest CPU baseline.

65
00:14:39,921 --> 00:14:59,147
[Dr. Ada Shannon] Right. Quicker ADC and PQ Fast Scan already use 4-bit codes with 8-bit lookup tables in SIMD registers. ADSampling and RaBitQ from Jianyang Gao and Cheng Long offer principled error bounds, while this predictor is empirical. Running those would show how much of the gap is hardware and how much is the idea.

66
00:14:59,147 --> 00:15:00,494
[Hal Turing] So who should care?

67
00:15:00,494 --> 00:15:21,903
[Dr. Ada Shannon] Anyone designing bandwidth-bound, billion-scale retrieval around near-memory HBM. You can reuse two ideas without new silicon: predicting precision at the cluster level, and storing data as bit-planes so lower precision means fewer bytes moved. Both need testing in software. Bit-planes also complicate index updates, and the system assumes a static corpus.

68
00:15:21,903 --> 00:15:23,250
[Hal Turing] What would you run next?

69
00:15:23,250 --> 00:15:35,742
[Dr. Ada Shannon] First the fixed-8-bit ablation on the same accelerator. Then an open reproduction of ANNA, a same-index ANSMET comparison, text-embedding datasets, and stricter recall targets with reranking in the loop.

70
00:15:35,742 --> 00:16:01,563
[Hal Turing] So, closing it out. The paper shows that a large share of cluster-level distance work can run below eight bits at small recall loss, on these two datasets and this recall target. It also shows a bit-serial near-memory design can turn that into speed and energy savings in simulation. The boundary is that nobody has measured how much of the gain is the precision itself.

71
00:16:01,563 --> 00:16:03,049
[Dr. Ada Shannon] That's right. Thanks for listening.

72
00:16:03,049 --> 00:16:04,999
[Hal Turing] Thanks, everyone. See you next time.