Click Tools > Internet Options > Advanced
Under Browsing, uncheck Display Accelerator button on selection
Click Apply and OK

How do I configure that via the registry?
[HKCU\Software\Microsoft\Internet Explorer\Services]

Note: Accelerators is not turned off. The setting just turns off the blue shortcut icon. Accelerators will still appear when you highlight and right-click text,