Windows Shell Extensions: Basics, Examples, and Common Problems | Apriorit (2024)

Home » Blog » Software Development Blog » Windows Shell Extensions: Basics, Examples, and Common Problems

Shell extensions are a powerful and flexible way to extend Windows Shell capabilities. However, when working with Shell extension handlers you can encounter hidden difficulties.

In this article, we describe the general approach to creating Windows Shell extensions based on the example of shortcut menu and icon overlay handlers. We also explain a number of pitfalls you may face when developing these types of extensions and offer several best practices to avoid them.

What are Shell extensions?

What is a Shell extension? To define Shell extensions, let’s look at the Windows Shell extension handler that allows you to extend the usual set of actions while working with Windows Explorer. Shell extensions can be represented as individual plug-ins to Windows Explorer. They can be used to add a new tab to the Properties window, change a file preview, and do other things.

Before taking any action, the Shell calls registered extension handlers to customize this action. A typical example of such adjustment is a Shell extension handler for the shortcut menu.

Depending on the file type, Shell extension handlers can be added either to all types of objects within Windows Explorer or only to certain types of objects.

Shell extension handlers used with specific file types:

Shell extension handlers that don’t depend on the file type:

However, no matter what file type you apply a handler to, using Shell extensions can slow down Windows Explorer. In addition to Windows Explorer, other programs including Dropbox, TortoiseSVN, WinRAR, and SkyDrive establish their own sets of Shell extensions.

Creating Shell extension handlers

In this section, we’ll discuss the process of creating Windows Shell extension handlers based on the example of overlay and context menu extensions. Other types of Shell extensions can be implemented in a similar way.

Each Shell extension handler is a Component Object Model (COM) object. Handlers must have their own globally unique identifier (GUID) and be registered before the Shell can use them. The registry path is determined by the type of extension handler. Now let’s go through all the stages of creating a Shell extension handler.

Implementing the required functions and interfaces

Since a Shell extension handler is a COM object, it’s implemented as a dynamic link library (DLL). At the same time, just as any COM object, a DLL must export the following standard functions:

• DllMain – Creates an entry point to a DLL
• DllGetClassObject – Gets an object using the Class factory
• DllCanUnloadNow – Calls a DLL before unloading to check whether it’s currently being used
• DllRegisterServer – Registers a COM object in the registry
• DllUnregisterServer – Removes the COM object entry from the registry

The ClassFactory is used to create the component object and must implement the IClassFactory interface. Here’s how the ClassFactory looks:

class ClassFactory : public IClassFactory { public: // IUnknown IFACEMETHODIMP QueryInterface(REFIID riid, void **ppv); IFACEMETHODIMP_(ULONG) AddRef(); IFACEMETHODIMP_(ULONG) Release(); // IClassFactory IFACEMETHODIMP CreateInstance(IUnknown *pUnkOuter, REFIID riid, void **ppv); IFACEMETHODIMP LockServer(BOOL fLock); explicit ClassFactory(); ~ClassFactory(); private: long m_cRef; };

In addition, the class that encapsulates the logic of the extension handler must implement the IUnknown interface, like all COM objects. Interfaces that are specific to different types of Shell extensions inherit from this interface. The IUnknown interface looks as follows:

class IUnknown{public: virtual HRESULT QueryInterface(REFID riid, void** ppv)=0; virtual ULONG AddRef () = 0; virtual ULONG Release() = 0;};

Note that several extension handlers can be implemented in a single DLL. At the same time, you should specify GUID, perform registering in the registry, and set up a ClassFactory for each handler (although you can use a common ClassFactory for several extension handlers).

Definition of interfaces specific to Shell extensions

In addition to the required IUnknown COM interface, extension classes must implement an interface that’s specific to Shell extensions. For different types of extensions, there are different sets of required interfaces.

Most extension handlers must also implement either an IPersistFile or IShellExtInit interface in Windows XP or earlier. These were replaced by IInitializeWithStream, IInitializeWithItem, and IInitializeWithFile in Windows Vista. The Shell uses these interfaces to initialize the handler.

For a shortcut menu handler, the interfaces used are IShellExtInit and IContextMenu, and for the overlay icon handler, it’s the IShellIconOverlayIdentifier interface. Let’s look closer at each of them.

IShellExtInit

The IShellExtInit interface provides only an extension handler initialization function:

class IShellExtInit : public IUnknown{public: virtual HRESULT Initialize(LPCITEMIDLIST pidlFolder, LPDATAOBJECT pDataObj, HKEY hKeyPro-gID) = 0;};

The Initialize method is called by Windows Explorer before you perform an action that this handler extends. For example, in the case of extending the context menu, this method will be called after clicking on selected files and before the context menu opens.

In the Initialize method, you can implement logic that filters extension work depending on the type of selected files or any other conditions. Furthermore, the list of objects, which the user chooses, can be obtained in the Initialize method.

Note that you can use the IShellExtInit interface only when you’re writing a handler based on the IContextMenu or IShellPropSheetExt interface, as other Shell extensions based on different interfaces use different initialization methods.

IContextMenu

The IContextMenu interface contains methods for working with the context menu:

class IContextMenu : public IUnknown{public: virtual HRESULT QueryContextMenu(HMENU hMenu, UINT indexMenu, UINT idCmdFirst, UINT idCmdLast, UINT uFlags) = 0; virtual HRESULT InvokeCommand(LPCMINVOKECOMMANDINFO pici) = 0; virtual HRESULT GetCommandString(UINT_PTR idCommand, UINT uFlags, UINT *pwReserved, LPSTR pszName, UINT cchMax) = 0;};

The QueryContextMenu method is responsible for adding new menu items to the context menu. This is where you can add logic for creating new menu items. A menu item can be created by the WinApi InsertMenuItem function.

Each menu item is linked to its ID. The idCmdFirst parameter is the minimum ID value that can be assigned to a menu item. Accordingly, idCmdLast is the maximum ID value. It’s important that if a menu item contains a submenu, the submenu items are assigned to IDs created after the ID of the menu item.

The QueryContextMenu method returns the difference between the first and last assigned ID plus 1.

QueryContextMenu is called by Windows Explorer only if the extension is registered for separate file types.

The InvokeCommand method processes clicks on added menu items. Each menu button is identified by the verb value, which is specified while creating the item in QueryContextMenu. Verb is a string with the command name that is executed by the menu item. Verb is given in two versions: with and without the use of Unicode. Therefore, the verb format should be checked before executing a command.

The GetCommandString method is used to obtain a canonical verb name assigned to the menu item. This method is optional and set for Windows XP and earlier versions of Windows. It’s used to display the hint text for the selected menu item in the status bar.

IShellIconOverlayIdentifier

This interface must be implemented by the overlay icon handler:

class IShellIconOverlayIdentifier: public IUnknown{public: STDMETHOD(GetOverlayInfo)(LPWSTR pwszIconFile, int cchMax, int *pIndex, DWORD* pdw-Flags) = 0; STDMETHOD(GetPriority)(int* pPriority) = 0; STDMETHOD(IsMemberOf)(LPCWSTR pwszPath, DWORD dwAttrib) = 0;};

The GetOverlayInfo method is used for the first call to the overlay handler in order to add an icon to the system’s list of images. After that, the icon can’t be changed and subsequent calls to GetOverlayInfo must return the path of the file containing the icon and the icon index. The icon index is the serial number of an icon (starting with 0) in the DLL resource list.

If the method returns an error for the first run, the icon will not be loaded, even if subsequent calls to GetOverlayInfo are successful. This can lead to a situation when the handler is working and the icon is not displayed.

The GetPriority method is used to specify the overlay icon’s priority over other icons. Depending on it, the handler can either be a top priority or not. The highest priority is 0.

The IsMemberOf method must contain logic that indicates whether to apply the icon to the given object in Windows Explorer. This method is called in turn for all objects in the current Windows Explorer window when the window is opened and after it’s updated. If you don’t specify any filters here and just return S_OK, the icon will be displayed on all objects in Windows Explorer.

Registering Shell extensions

The logic for registering Windows Shell extensions must be implemented in the DllRegisterServer function. This function is called when installing a Shell extension and is responsible for COM object registration. Accordingly, the DllUnregisterServer function must contain code for deleting records from the registry.

Note that you may receive an 0x80070005 error when you try to register a DLL on Windows XP or Windows Server 2003. This happens because their strict security schemes allow only admins to initiate DLL registration.

The registry hive, which contains information about extensions, differs depending on the type of extension handler. For example, for the shortcut menu handler, the path is as follows:

HKEY_CLASSES_ROOT<file_type>shellexContextMenuHandlersExtensionName

The <file_type> parameter may be defined:

• For a specific file type: BMP, DOCX, PNG
• For all file types: *
• For a group of objects: Drive, Directory, Folder, LibraryLocation

The shortcut menu handler must be registered separately for each object type.

The icon overlay handler is registered once using the path:

HKEY_LOCAL_MACHINESoftwareMicrosoftWindowsCurrentVersionExplorerShellIconOverlayIdentifiersExtensionName

In addition, you need to register the COM component itself:

HKEY_CLASSES_ROOTCLSID<COM_object_GUID>InprocServer32ThreadingModel

For example, the DllRegisterServer function, which registers overlay icon handlers and shortcut menu handlers for all file types (“*”), might look like this:

// GUIDs for COM objectsconst CLSID kMenuExtCLSID = { 0xa1239638, 0xb2ba, 0xa2c8, { 0xa4, 0x3b, 0xf6, 0xe2, 0x10, 0x81, 0xf, 0x77 } };const CLSID kOverlayExtCLSID = { 0xa4569638, 0xc2ba, 0x42c8, { 0xb4, 0xfb, 0xc6, 0xe6, 0x10, 0x91, 0xf, 0x23 } };STDAPI DllRegisterServer(void){ HRESULT hr; wchar_t szModule[MAX_PATH]; if (GetModuleFileName(g_Dll->DllInst(), szModule, ARRAYSIZE(szModule)) == 0) { hr = HRESULT_FROM_WIN32(GetLastError()); return hr; } // register COM object for shortcut menu handler hr = RegisterInprocServer(szModule, kMenuExtCLSID, L"AppCustomExtensions", L"Apartment"); if (SUCCEEDED(hr)) { // register shortcut menu handler hr = RegisterShellExtContextMenuHandler(L"*", kMenuExtCLSID, L"ContextMenuExt"); } // register COM object for overlay icon handler hr = RegisterInprocServer(szModule, kOverlayExtCLSID, L"AppCustomExtensions", L"Apartment"); if (SUCCEEDED(hr)) { // register overlay icon handler hr = RegisterShellExtOverlayIconHandler(kOverlayExtCLSID, L"OverlayIconExt"); } return hr;}

The RegisterInprocServer, RegisterShellExtContextMenuHandler, and RegisterShellExtOverlayIconHandler functions contain the code for adding new hives and keys to the registry.

Preparing resources for icon overlay handlers

Icons must be prepared to be correctly displayed by Shell extension handlers.

For the context menu, we need a BMP image. Overlay handlers are a bit more complicated: the icon must be in ICO format and support the following dimensions:

• 10×10 (for 16×16)
• 16×16 (for 32×32)
• 24×24 (for 48×48)
• 128×128 (for 256×256)

Overlay handler icons will be displayed in the lower left corner of the main icon and should take up no more than 25% of the file icon (except for 10×10 for 16×16).

When scaling objects in Windows Explorer, the right size will be determined authomatically.

Installing Shell extensions

Shell extensions are installed using the standard Regsvr32 utility. For example:

> regsvr32 C:ExampleShellExtensions.dll

When you run this command, the DllRegisterServer function is called, and this function provides extension registration in the registry. All the logic for adding and removing entries from the registry must be implemented by the developer.

After this, the shortcut menu handler immediately begins its work, which can be verified by opening a menu. As for the overlay handler, the icons are not going to appear immediately after registration. In order to download the required resources, you must relaunch Windows Explorer.

Removing Shell extensions

For removing a Shell extension, you can use the same approach used for its installation — namely, Regsvr32:

> regsvr32 /u C:ExampeShellExtensions.dll

Meanwhile, when we call the DllUnregisterServer function, the system deletes records from the registry.

After removing the overlay handler, you’ll also need to relaunch Windows Explorer.

Common problems and their solutions

You may face some Windows Shell extension problems when writing Shell extension handlers. Some of these problems are listed below together with ways to overcome them:

1. Overlay icons are not displayed

The problem: As described above, this problem may be in the logic of the GetOverlayInfo function. Also, as Windows uses only a limited number of registered overlay icon handlers, all other handlers are simply ignored.

A list of the handlers that won’t be ignored is determined by the name in the registry branch: HKEY_LOCAL_MACHINE Software Microsoft Windows CurrentVersion Explorer ShellIconOverlayIdentifiers. The system takes the first 15 overlay icon handlers in alphabetical order. Therefore, if Dropbox or TortoiseSVN, which record a number of extensions, are installed in the system, they can become a reason for not displaying icons.

How to solve this: Add any character before the overlay handler name during registration. This will make the extension appear at the top of the list of extensions. For example, you can add spaces before the handler name.

2. Issues with displaying icons with transparent backgrounds in the context menu

The problem is in using a BMP image that does not support transparency while you add a new menu item. If you use ICO or PNG files, then the icon will be displayed on a black background after its conversion.

How to solve this: Redraw the ICO to BMP format in the code while the extension is running. For example, the following implementation can be used (with UX-THEME.DLL):

m_beginPainFunc = (BeginPaintFunction)::GetProcAddress(m_hUxTheme,"BeginBufferedPaint");m_endPaintFunc = (EndPaintFunction)::GetProcAddress(m_hUxTheme,"EndBufferedPaint"); ... SIZE iconSize; RECT iconRect; HBITMAP hBmp = NULL; HDC hdcDest = CreateCompatibleDC(NULL); // set the icon size iconSize.cx = GetSystemMetrics(SM_CXSMICON); iconSize.cy = GetSystemMetrics(SM_CYSMICON); SetRect(&iconRect, 0, 0, iconSize.cx, iconSize.cy); if (hdcDest) { if (SUCCEEDED(CreateHbitmap(hdcDest, &iconSize, NULL, &hBmp))) { HBITMAP oldBitmap = (HBITMAP)SelectObject(hdcDest, hBmp); if (oldBitmap) {// set the necessary parameters BLENDFUNCTION bfAlpha = { AC_SRC_OVER, 0, 255, AC_SRC_ALPHA }; BP_PAINTPARAMS paintParams = {0}; paintParams.cbSize = sizeof(paintParams); paintParams.dwFlags = BPPF_ERASE; paintParams.pBlendFunction = &bfAlpha;// repaint the icon HDC hdcBuffer; HPAINTBUFFER hPaintBuffer = m_beginPainFunc(hdcDest, &iconRect, BPBF_DIB, &paintParams, &hdcBuffer); if (hPaintBuffer) { if (DrawIconEx(hdcBuffer, 0, 0, hIcon, iconSize.cx, iconSize.cy, 0, NULL, DI_NORMAL)) { ConvertBufferToPARGB32(hPaintBuffer, hdcDest, hIcon, iconSize); } m_endPaintFunc(hPaintBuffer, TRUE); } SelectObject(hdcDest, oldBitmap); } } DeleteDC(hdcDest); }

3. Issues with displaying the context menu

There can be several problems associated with the context menu.

The problem: After registering a shell extension for LibraryLocation, the context menu extension is not displayed.

The problem may be in the implementation of the Initialize method in the IShellExtInit interface, which is called to initialize the Shell extension. IdataObject, which is used for receiving objects for which the extension will be displayed, is passed to the Initialize method as a parameter. So when you click on the library folders (Documents, Music, etc.), any attempts to receive the CF_HDROP format objects from it don’t lead to the expected result.

How to solve this: Use the CFSTR_SHELLIDLIST format, which is similar to CF_HDROP but contains PIDL (a pointer to the object ID in the ITEMIDLIST Shell structure) instead of the file path. It allows CFSTR_SHELLIDLIST to process system and virtual file system objects.

Int m_shellIdList = RegisterClipboardFormat(CFSTR_SHELLIDLIST); IFACEMETHODIMP ContextMenuExt::Initialize(LPCITEMIDLIST pidlFolder, LPDATAOBJECT pDataObj, HKEY hKeyProgID){ if (pDataObj == NULL) { return E_INVALIDARG; }  HRESULT hr = E_FAIL;  // instead of // FORMATETC etc = { CF_HDROP, NULL, DVASPECT_CONTENT, -1, TYMED_HGLOBAL }; FORMATETC fe = { (CLIPFORMAT) m_shellIdList, NULL, DVASPECT_CONTENT, -1, TYMED_HGLOBAL }; . . .}

The problem: When registering a Shell extension for AllFileSystemObjects, clicking .lnk files adds this Windows Shell extension to the context menu twice.

The problem is that the extension works both for the object and for the link that points to it.

How to solve this: In the QueryContextMenu method of the IContextMenu interface, not only must the CMF_DEFAULTONLY flag be checked but also the context menu extensions should ignore objects with the CMF_NOVERBS and CMF_VERBSONLY flags.

The problem: The context menu extension is not displayed for some types of files, although it can be registered for all types.

The problem may be in the implementation of the QueryContextMenu method of the IcontextMenu interface. In case of successful implementation, it should return HRESULT with a code value equal to the maximum offset of the command identifier that was assigned plus one.

For example, if idCmdFirst equals 5 and 3 more menu items with 5, 7, and 8 command identifiers were added, then the return value should be: MAKE_HRESULT(SEVERITY_SUCCESS, 0, 8 – 5 + 1).

If you forget to add one, the context menu extension will still work but not in all cases.

How to solve it: Make sure the QueryContextMenu method of the IcontextMenu interface is successfully implemented.

4. The problem with starting a thread from DllMain

The problem: When you start a thread in DllMain (for example, to initialize something), there may be problems with DLL unloading using FreeLibrary.

For example, a DLL with Shell extensions loads dynamically in order to unregister extensions manually. At the same time, you start a thread in DllMain to initialize logging. In this case, the DllUnregisterServer is called and successfully unloads the DLL. Then, as you might expect, the system calls FreeLibrary.

In this example, it may happen that a thread called from DllMain either has not completed execution or has not started yet, while a DLL has already been unloaded from the process context. As a result, memory areas are damaged and access violations may occur at any time when there’s a reference to these areas.

How to solve it: Wait for the end of the initialization thread in the exported DLL functions.

Conclusion

In this article, we discussed all types of Windows Shell extensions as well as a general approach to their implementation. In addition, we described the basic steps for creating a shortcut menu and overlay icon handlers. We also examined several pitfalls of implementing these extensions and provided tips on how to overcome them. To get the sample project sources, you can initiate the Windows Shell extension download here.

Apriorit has a team of qualified Windows system developers who can assist you with a Windows project of any complexity. Get in contact with us using the form below!