My contribution is not meant to enlighten others or proclaim any particular truth, rather to offer to the collective conscience my perceptions of what I've observed over time - right or wrong:
I use a propane torch in dim light, with a cordless drill/deep-well socket, dump into water to stop the heat, count seconds in my head;
I tend toward milder cartridges to begin with; 30-30, 222, 7x57, 6.5x55, 257 Roberts,.....
I tend toward working my brass as little as possible;
I've never annealed straight-wall pistol brass;
I've annealed brass at the 3x-fired mark, or thereabouts, except that I often forget and just anneal when I can't remember when I last did or when I find a split neck;
I usually anneal just before or right after fire-forming when necking up or down, or like making 300 BLK brass before commercial was available.
What has stood out among many other less notable annealing sessions:
In the early to mid nineties, I came across relatively new, factory Winchester ammo (garage/estate sales, bargain bins at guns shops) on which every single neck split upon the first firing. Point: your methods/you are not always the source of the problem. I don't hold a grudge, but I have not bought Winchester brass since. Right or wrong, I might add.
AMMONIA is NOT GOOD for BRASS! DO NOT put Brasso in your tumbling media! My dad and I shared everything back when we couild shoot together regularly. His brass was my brass, my bullets wee his bullets, boxes and bags got passed back and forte and no one remembered what belonged to which one of us. Didn't matter. Didn't matter until I found out he was using Brasso in his tumbler! I and STILL finding ammonia-fouled brass thirty years later. Until I figured this one out, I thought it was a lack of annealing issue.
I agree with whoever said "time" has an affect on brass. Can't explain it, but I have had cases split just sitting, waiting to be used. OK, some of the cases are thirty and forty years old. SOME not ten years old, and with green stuff growing out of the crack and on the basses of bullets.
Going through my dad's stuff recently, I found a bunch of loaded 6.5-284 loads from 1994. Second firing, no annealing (Winchester 284 brass, necked down to .264") and over half the necks were split neatly as could be from mouth to shoulder. This brass was fired ONCE after necking down, neck sized after and loaded. Necks were .261" ID with Nosler .264" bullets loaded. I am absolutely positive that these were not tumbled in Brass-infected media. Should they have been annealed? I've fire-formed 7x57 AIs, necked 243s up to .257" with no issue and no annealing. Maybe just a less tolerant batch of brass?
What these other neck split issues have to do with annealing is that they all called into question the need for annealing, but may not even have been related. I don't know for certain that I've personally ever HAD to anneal, but I see it so rt of like changing your oil - you don't necessarily SEE the benefit as you do it, but it SHOULD have benefit over time.
Annealing makes me feel good about my brass and I do SO love the "look," as dumb as that may sound.