From: "Matthias Paul" Organization: Rechenzentrum RWTH Aachen To: opendos AT delorie DOT com Date: Mon, 9 Oct 2000 16:51:08 +0100 Subject: architecture of DOS disk sub-system [Re: more than 8 GB] X-mailer: Pegasus Mail v3.22 Message-ID: <3F83BA00F21@reze-1.rz.rwth-aachen.de> Reply-To: opendos AT delorie DOT com On Thu, 28 Sep 2000, Florian Xaver wrote: > >Given the memory footprint of the shipping issue (with > >EMS and DPMS support disabled) and the fact that it is not > >cached by NWCACHE, this is not actually a solution. > > Why isn't it cached? Why can't nwcache cache it? I think, this > is a "bug", because also smartdrv support caching a redirector. Florian, you may call this a missing feature, but it definitely is not a bug. Caching volumes accessable through INT 13h and caching drives hooked into the system via the redirector interface are rather different tasks. To understand the differences and the various implications, I have to explain some details of the hierarchical driver model of the DOS file and disk sub-system: When DOS gets called to create/delete/open/read/write etc. a file or a directory (via its qualified filespec), it will call the redirector interface to inform external components of the request. If an external driver hooking the redirector (like a network redirector, a CDEX, or DRFAT32) detects that the request addresses one of the logical volumes handled by that driver (e.g. by checking the drive letter in the filespec), it will perform the function by itself (see below). When the request is not handled by any such external component, it is processed by the internal file-system logic built into DOS (which, of course, only supports the FAT12/16 file-systems). All the pathspec crunching, directory navigation, FAT allocation etc. is handled at this level. On the lower end of the FAT12/FAT16 file-system logic, any file operatings are translated into read/write requests via the INT 25h/ INT 26h "Absolute disk read/write" interface. At this level, volumes are addressed in terms of *logical* sectors (counted from 0 upwards, depending on the size of the partition) on *logical* volumes (e.g. 0 = A;, 1: = B, 2 = C:, ...). The size of these sector buffers is 512 bytes. The DOS BIOS (IBMBIO.COM), which implements the low level disk driver, will transform these requests into requests to the *physical* medium, taking into account the relative starting position of the corresponding partition on the physical (hard)disk, translating the logical addressed sectors into the traditional Cylinder/Heads/Sectors (CHS) addressing scheme (this time *physical* sectors a 512 bytes) and pass this down to the ROM BIOS INT 13h level, which handles *physical* media (e.g. 00h = 1st floppy, 01h = 2nd floppy, 80h = 1st harddisk, 81h = 2nd harddisk). Finally, the ROM BIOS actually performs the physical read/write operations on the disk by directly accessing the hardware. For native FAT32 drives (MS-DOS 7.10+ and OEM DR-DOS 7.04+), this works similar to FAT12/FAT16 except that DOS uses INT 21h/7305h "Extended Absolute Disk Read/Write" instead of INT 25h/INT 26h, and that the DOS BIOS will issue CHS or LBA calls at the INT 13h level (MS-DOS up to including 7.0, and DR-DOS up to 7.03 will only issue the traditional CHS calls). This staggered architecture and in particular the abstraction level of the redirector interface allow to "plug" foreign filesystems into DOS, even for media which must not necessarily be available at the INT 13h level. For example, a network driver (using the redirector interface like PNW's VLM) would intercept attempts to access network drives and emulate the filesystem while the requests would actually be tunnelled through the network protocol stack, transmitted through the network hardware and may then touch the disk (or other media) on a remote machine (for example using a non-FAT-filesystem on the remote server). The redirector interface also allows to plug in filesystems like HPFS, NTFS, ext2fs, FAT32 (and virtually anything you like) into DOS, as long as these filesystems can be presented in a manner used to access files under DOS (that is using the well-known filespec naming conventions). Depending on possibly advanced file-system concepts you may not be able to make use of all its features under DOS, however, you *are* able to access files and navigate in directory trees as if the volume would be natively supported by DOS. The main problem with drivers hooking the redirector interface is that they usually are memory hogs, because they have to implement the whole filesystem logic (and more), and usually advanced filesystems like HPFS, NTFS, or ext2fs are designed for speed and not for minimal resource footprint. (A solution to the memory footprint problem could be DPMS, Cloaking, etc.) FAT32 is kind of an exception, because on one hand it is much more that just a FAT filesystem with 32-bit placeholders in the FAT, but on the other hand it is still not *that* different from FAT12/FAT16, that parts of the FAT12/FAT16 code in the kernel could not be reused. This is the reason, why native FAT32 makes sense (although it would occupy ca. 3- 5 Kb of resident memory, and internal data structures will have to change for FAT32, making the kernel incompatible with DOS before 7.10), but native HPFS, NTFS, etx2fs etc. are simply illusoric under DOS. CDEX drivers like NWCDEX or MSCDEX use the redirector to plug in the ISO 9660 filesystem, as it is used on CD-ROMs. At the lower level of the ISO 9660 file system logic, the driver cannot use the INT 25h/ INT 26h level, because CD-ROMs use 2048 (and more) bytes per sector, not 512 bytes. For the same reasons, these lower levels would not be able to handle CD-ROM drives because they are not usually represented at INT 13h (the exceptions are "El Torito" emulation modes in conjunction with bootable CDs). This is why NWCDEX/MSCDEX directly talks to a CD-ROM specific device driver (/D:OEMCD001 option). While these device drivers are used to access drives, they still cannot be original block device drivers (represented by drive letters - see MEM /A) because of the incompatible nature of CD-ROMs. Instead, they install themselfs as character device drivers (e.g. "OEMCD001"), but they are only dummy character device drivers having a special signature in their header that identifies them as *CD-ROM* device drivers. NWCDEX verifies that this signature is OK before it attempts to communicate with the driver by issueing CD-ROM device driver specific commands. The CD-ROM device driver handles all the hardware dependent stuff: A generic CD-ROM device driver like ATAPI.SYS would directly access the CD-ROM hardware on the corresponding IDE channel. Older proprietary CD-ROM hardware required proprietary hardware- specific drivers, e.g. the MTMCDAE/MTMCDAS.SYS drivers were used for Mitsumi FX001 double-speed CD-ROMs back in the early 90th. In the modular SCSI APSI architecture, a CD-ROM device driver like SYMCD.SYS (for Symbios/LSI based host adapters) would pass the request down to the ASPI manager (e.g. ASPI8xx.SYS for the SYM8xx family of SCSI chips), which would physically access the SCSI controller to talk to the SCSI CD-ROM drive. Well, finally back to DRFAT32.EXE: This is very similar to NWCDEX, but it implements a different file- system: Not ISO 9660, but FAT32. At the lower end it talks to INT 21h/7305h to provide some means for caching, but hooks these functions to get called back, so that the requests can be transformed into calls to DRFAT32.SYS, a device driver that installs as character device driver, but provides a number of DRFAT32 specific functions (see INTER61: INT 21h/4402h"DRFAT32" and INT 2Fh/0802h for details). The architecture is very similar to the CD-ROM device driver + CDEX redirector model, but the implementation and the device driver commands are unique (in fact early ALPHA issues of DRFAT32.EXE fully emulated a CDEX, while the shipping DRFAT32.EXE 1.00 is just a redirector, not a CDEX). Depending on the ROM BIOS support for LBA, the used partition types, and DRFAT32.SYS configuration switches, DRFAT32.SYS will translate/remap between CHS and LBA addressing schemes and then call the ROM BIOS INT 13h. > BTW: I have some questions to redirectors. Maybe you can answer it > quickly: > Is there any documentation on the web, how to make a redirector? The redirector interface is not documented. The book "Undocumented DOS" (2nd edition, Addison & Wesley) by Andrew Schulman et al is the definite source. Search for the "Phantom RAM disk" in there. Microsoft has documented the MSCDEX 2.1 interface (but only the CDEX API, not the redirector API, and not most of the peculiarities of the MSCDEX implementation, which make it so difficult to design a CDEX being 100% compatible with MSCDEX). This should be available on SimTel and other places. Search the sub-directories holding CDROM drivers for something like MSCD*.*. (Sorry, I don't have a link right now.) For more accurate and up-to-date info always refer to Ralf Brown's "Interrupt List": Download the current issue INTER61, and search the INTERRUP.? files for "CDEX" (INT 2Fh/15xxh), "redirector" (INT 2Fh/11xxh), and "DRFAT32" (e.g. INT 21h/4402h and INT 2Fh/0802h). This will give you a very good overview on the required functionality and some implementation details (not on file system internals, of course). > Why have all redirectory a sys driver and an exe file > (like atapi.sys and nwcache or drFAT32.sys and drFAT32.exe)? See above. > Why it is so slow? Or isn't is so slow, only nwcache doesn't support > redirectors? Volumes handled by redirectors are not necessarily slower than volumes handled by built-in drivers (if not taking the amount of optimization into account). It just seems to be so, because redirector drivers usually access relatively slow media like networks, CD-ROM devices, or complex file systems like HPFS, NTFS, or FAT32. However, it is not possible to cache at the redirector level itself, so you would need a cache, that intercepts one of the lower levels. For network drives, this could be some kind of packet buffering in the protocol stack, for CD-ROM drives this is the device driver level. For the DRFAT32 driver, this may be INT21/7305h (or INT 13h). (At the INT 13h level, the cache must be specifically designed to look for drives controlled by DRFAT32.SYS since the usual methods to retrieve the information about the drive obviously don't work for DRFAT32.SYS. This would be easy, but it needs special handling in the cache's initalization code, so none of the currently existing caches will do this. Also, the cache would need to handle both, LBA and CHS. Hence the INT 21/7305h level is preferrable for DRFAT32.) Similar to NWCDEX.EXE, DRFAT32.EXE itself implements a rudimentary cache below the file system logic (/M:nnn). NWCDEX handles approx. 2Kb sized CD-ROM sectors, but DRFAT32.EXE handles 512 byte sector buffers. The total size of this buffering is 60 Kb (/M:120 for DRFAT32.EXE), which is "almost nothing" for FAT32 drives, which usually have very large File Allocation Tables (FATs) per se. NWCACHE and (pre-MS-DOS 6.2 issues of SMARTDRV) cache drives at the ROM-BIOS INT 13h CHS level, not at the higher INT 25h/INT 26h or the block device driver level. SMARTDRV 4.2+ of MS-DOS 6.20+ seems to cache at the INT 25h/INT 26h "Absolute disk read/write", and AFAIK at the CD-ROM device driver level, when it caches CD-ROM drives. At least, some 3rd party stand-alone caches for CD-ROMs (like CDQUICK) for sure cache at the CD-ROM device driver level. I don't know, if SMARTDRV of MS-DOS 7.10+ (Windows 95 OSR2 and later) will cache FAT32 drives at the INT 21h/7305h level. If it would, this may allow to cache DRFAT32 drives as well. I just never tried. Someone willing to test this out? Matthias ------------------------------------------------------------------- Matthias Paul, Ubierstrasse 28, D-50321 Bruehl, Germany eMail: Web : http://www.rhrz.uni-bonn.de/~uzs180/mpdokeng.html -------------------------------------------------------------------