Windows 7 / Windows 2008 R2 I/O Subsystem Logical View


Windows I/O subsystem manages and provides interface to hardware devices for several applications and for Operating System. The design goals of Windows I/O system is to provide device abstraction ( DMA/ hardware abstraction layer/ bus drivers) for hardware and software components

Below is the logical view Windows I/O manager functions.

Windows IO function Logical diagram

Advertisements

Remote Debugging Using Namepipe


Introduction:

How to use debugging tools to perform remote kernel debug?. This document explains the debuggers used and the steps needed to attach and perform live debugging. Windows Operating system consists of 2 modes User mode and Kernel mode. User mode applications such as Wordpad , Notepad , managed applications , unmanaged applications executes at User mode and Operating system core components such as File system drivers, Network stack , Interrupt controllers , Memory management, Graphics , GDI are managed at kernel level.

There are many instances where an Engineer has to perform live debugging on a non-responsive operating system or debug an application remotely to understand the interaction between user application and kernel mode subsystem or driver. This article will help Engineers to perform live debugging from Host to the Target using Namepipe

 

Below article explains the steps involve to successfuly perform remote debugging on a VM.

User Mode To Kernel Mode Switching


Windows operating system is divided into user mode and kernel mode space. Applications can run either in user mode or in the kernel mode in the form of a driver.  Applications running in user mode have huge stack space and most of the times drivers are implemented in kernel mode.

The applications can communicate to the drivers either

  • Calling a dynamic link library ( dll )
  • Calling a win32 API

 

When an application call one of the routine exposed by the dynamic link library , the dll in-turn communicates with win32 API and then the API calls the appropriate kernel mode subroutines . When we break down more in-detail , win32 API calls kernel mode client driver which communicates to class drivers provided by Microsoft.

When performing application testing , test engineers have to consider

  • Application interfaces being debugged
  • Memory usage
  • Operating system performance
  • Calls made to the kernel mode drivers
  • Context switch rate

There might be chances that application is waiting for the instruction to complete at the driver end , but the engineer end up debugging the application

 

Memory Management – Part 1


Programs cannot access physical memory directly, they have to access memory using the below techniques.  And most important all the below memory models has Code , Data and Stack segments. Linear address space is virtual address which is mapped to the physical address.  

a) Flat Memory Model  

b) Segmented Memory Model  

c) Real – Address Mode Memory Model.

 

Some of the interesting facts are  

-IA-32 and IA-64 processors uses either Flat Memory Model or Segmented Memory Model. 

-Real Address mode uses segmented memory of 64 KB blocks.  

-During conversion of linear address to physical address, the addresses are sent out on processor address bus.

Wait for more 🙂

Analyzing Windows Process -NotMyFault


I was trying to generate dump manually on my Dell D630 laptop which is running on 32-bit architecture with 2GB Ram , 1.5 times page file set. I understand the different ways to generate complete / kernel / mini dumps. But I was interested to run Mark’s Notmyfault tool to generate complete memory dump , you can find more information about the tool from the link 

One of the observation I made when trying to set specific process as implicit is that windbg throws the following error “Process <ID> has invalid page directories”. This behavior you observe for any process I try to make it implicit.  

Why does Windbg thinks page directories are invalid ?, page directory is process specific and has the information about virtual memory to physical memory mapping, so I would think some of the pages might be missing when capturing dump ? , I might be wrong , but I was able to successfully see the thread stack. 

 

Lab : 

Step 1 : Download notmyfault tool , and manual crash the system to generate complete memory dump 

Step 2 : launch Windbg with symbols loaded appropriately and attach the dump  

Step 3 : try running .process <ID> which will result in above message  

Step 4 : try reloading the symbols for the application using .process /r /p  <id >  

Step 5 : Now run !process <ID / address> or !process 0 0 <process name > 

Step 6 : Navigate to the thread you desire to debug and run .thread <id / address>  which would say “implicit hread is <ID>”

Step 7 : Run KV to examine the thread stack.

 

 

Analyzing Windows Process Memory


Addresses are always virtual addresses, whether the process is running under User Mode or the process is running under Kernel Mode.

Its has been more theoretical experience of understanding process data, but what happens under the hood is interesting. This document falls back to my previous post of “Basic Postmortem Of Windows Process” , once you have collected the dump , your next step is to identify the root cause of the problem, for which you need to identify the process and the thread activity accordingly.

Once you have identified the process, you should dwell down to the memory reserved for the process. Processor at the other end, addresses only Physical Memory , so there is a mapping between Virtual Memory and Main Memory. Intel architecture provides 3 memory models

  • Flat Memory Model
  • Segmented Memory Model
  • Real-address Memory Model.

 

Flat Memory Model : This model is also known as Linear address space, programs / applications view memory as one contiguous memory block  and it is a “byte” addressable.

 Segmented Memory Model : Memory is presented to application / program as segments , each segment stores code , data and stack and these segments are then mapped into linear address space.

Real-Address Memory Model: This was specifically designed for 8086 processor to provide support for programs written on 8086 architecture.

Alright!, enough of theory and let us now see how to view these addresses. As I mentioned above , debugger interprets addresses as virtual , and Windbg provides a way to map the virtual memory address to physical memory and vice versa using the below command. 

  • !ptov
  • !vtop

!ptov: provides a physical-virtual memory map for any given process. After identifying the process which you need to debug under the kernel debug mode, you should now select the page frame number of the process.

In order to analyze physical to virtual mapping , you should get the relevant address for which you need to use the address specified at DirBase attribute which is also Page Frame number. 

Eg: !ptov b340960   

Eg2 : If the Dirbase address is 0b340000 or 0b300000  , truncate the trailing 0’s and issue the command as  !ptov b34 or !ptov b3 

 

!vtop: To convert virtual to physical address , you can use this command. The syntax is different from !ptov , you need to provide virtual address as 3rd parameter . So first we need to get the virtual address of the process and then execute the !vtop command. 

In order to get the virtual address of the process you have to use !address command. You can run !address command either in user mode or in kernel mode, but I prefer running the command in the user mode context. Once you get the virtual address of the process you can use the command as below 

Eg: !vtop b34 00411000 where b34 is PFN and 00411000 is the virtual address.I will be updating the document with the ways to analyze the segmented / linear address space for an particular process

In order to get the virtual address of the process you have to use !address command. You can run !address command either in user mode or in kernel mode, but I prefer running the command in the user mode context. Once you get the virtual address of the process you can use the command as below 

Eg: !vtop b34 00411000 where b34 is PFN and 00411000 is the virtual address. 

You can also obtain the virtual address information of specific module using lm command shows the start and end address range of an process.

 

I will be updating the document with the ways to analyze the segmented / linear address space for an particular process.

 

Basic Postmortem Of Windows Process


PART I. ( Understanding Tools To Be Used )

 Every application  / driver comprises of processes and threads. Every process has the following 

  • Virtual Address space
  • Executable code
  • One executable thread ( MUST )
  • Open handles to objects
  • PID  (process ID )
  • Working sets
  • Security Context

 How do we view this and what are the tools required to view this data will be covered below.

  

Utilities Used To Debug Applications And Drivers:  

My article is specific to debug Microsoft applications and drivers written for Windows operating systems. Following are some of the utilities that every developer should know and use.

  • Visual Studio ( All versions )
  • ADPlus
  • Windbg
  • DebugDiag
  • BoundsChecker ( compuware )
  • Purify ( IBM )

  

Points To Be Noted: 

a)       Applications developed using .NET can take advantage of DebugDiag utility and then use Windbg side-by . 

b)       Driver developers can rely on Windbg. There was another classic tool “softice” which is no longer supported 

c)       Applications developed in MFC / ATL / C++ / C# can take advantage of Visual studio inbuilt debug functionality  and can use windbg side-by.

 

 

PART II. ( Preparing Debugging Environment )

 
Configuring Windbg Utility: 

One of the best utility used by Microsoft / debuggers to debug applications / drivers is Windbg. You can download windbg from the following website  http://www.microsoft.com/whdc/devtools/debugging/default.mspx . You can either download 32bit / 64bit installable. After downloading the utility, we have to configure it accordingly. Microsoft always assists newbie’s to use “help file” which exposes all the relevant documentation required to understand their product. Following are the immediate steps required once you have installed windbg utility.

Configuring Symbol File Path:  

This is the next immediate step after you download and install the Windbg utility. Symbol files contains the following information

  • Function names and addresses
  • Global variables
  • Local variables
  • Frame pointers

All of the above are bundled into a file called .pdb .These .pdb files are generated when you execute the program ( specifically the linker creates these .pdb files ). The location of these .pdb files are located in application’s project directory 

Eg:  c:\DocumentsandSettings\Administrator\MyDocuments\VisualStudio2005\Projects\debugcrash\debug\debugcrash.pdb

The above example belongs to the applications you develop in-house, but in-order to download the symbol files belongs to Microsoft , you need to provide the symbol file path which is provided by Microsoft  

Eg: SRV*your local symbol folder*http://msdl.microsoft.com/download/symbols  

Symbol file path should be configured under Windbg using following procedure

a)       Open windbg

b)       Navigate to File à Symbol File Path à provide the location accordingly

IMPORTANT NOTE: When debugging applications related to third party / applications for which you do not have the source code , you need to request the vendor for the .pdb files. 

  

Configure Source File Path: 

Once you have configured the symbol file path, you should configure the path for your source ( provided you are debugging application / driver which you have developed ). You can do this by navigating to File à Source File Path and provide the location of your executable. (follow image below )

  

  

PART III ( Analyzing The Application )

 Once you have configured the above settings, its time for us to analyze our application. The final step required is to hook the exe to Windbg which needs to be debugged. You have to open executable using Windbg through windbg à OpenExecutable à provide the path accordingly ( follow the image below )

*Please save the workspace when prompted.

Next step is to reload the symbol using .reload /f command, observe for any errors and resolve them accordingly. There is another way to reload the symbol files , by navigating File à Symbol File Path à reload . After you reload the symbol files, you have to confirm if the symbol files are loaded, this can be done using lm command , which gives you the .pdb files which are loaded by debugger ( follow the screen below )

When you hit lm command , you might see deferred status for modules . Debugger loads the symbols when required so you would see deferred status, in which you can force the symbol to load using .reload /f .

Till now , we have prepared the debugger , configured the symbol files, attached our exe to the debugger.  If you are debugging Your application , you can always put the relevant breakpoints to the functions you require, this depends on scenario / behavior of the problem. For this article, I would like to restrict my lab to view process information. For my lab purpose , I have written an MFC application called “Threadtest” with single parent process and single thread.  

To view the number of processes in your application, you should use pipe command |* which displays the below information.

My application is using one process and observe the ( dot ) beside ‘0’ which means this process is the current process.

Once we identify the number of processes for an application you are debugging / for driver  , you should know more about how the process is built , which includes the different data structures involved , Memory information, Heap, Environment etc.. One of the limitation of Windbg is that it doesn’t provide a way to “step-in” between user mode and kernel mode. There are multiple ways to tackle this behavior

  • Launch two debuggers , one in user mode and other in kernel mode and analyze the process or thread activity accordingly. 
  • Setup Target and Host and debug the process accordingly ( You can look into the help file in windbg how to setup Target and Host scenario )

 

Why do we need to perform step in to kernel mode ?, cant we analyze the data under user mode context ?

Answer to the above question is NO. In order to analyze the data structures responsible for constructing a process can only be analyzed in kernel mode.

 So what are the options available for any user under User Mode ?

Specific to process ( excluding the stack ) , the options provided by windbg is lean. It provides you to analyze

  • Number of processes of your application — | , | *
  • Active process — | .
  • Process which caused exception — | #

  

In order to analyze the Eprocess / PEB of the process / virtual memory of the process , you can perform the same in KD session. But I tried using !peb <PEB address of a process> under the user mode session and I was able to get the information which was similar running !peb <PEB address of a process> under LKD / KD.

 Eg: under user mode running  !peb 7ffdc000 will give you below information

     !peb 7ffdc000

    PEB at 7ffdc000

    InheritedAddressSpace:    No

    ReadImageFileExecOptions: No

    BeingDebugged:            Yes

    ImageBaseAddress:         00400000

    Ldr                       002524b0

    Ldr.Initialized:          Yes

    Ldr.InInitializationOrderModuleList: 00252568 . 00253048

    Ldr.InLoadOrderModuleList:           002524f0 . 002530f8

    Ldr.InMemoryOrderModuleList:       002524f8 . 00253100

    Base TimeStamp                     Module

    Under kernel mode !peb 7ffdc000

    !peb 7ffdc000

    PEB at 7ffdc000

    InheritedAddressSpace:    No

    ReadImageFileExecOptions: No

    BeingDebugged:            Yes

    ImageBaseAddress:         00400000

    Ldr                       002524b0

    Ldr.Initialized:          Yes

    Ldr.InInitializationOrderModuleList:  00252568 . 00253048

    Ldr.InLoadOrderModuleList:             002524f0 . 002530f8

    Ldr.InMemoryOrderModuleList:         002524f8 . 00253100

This article explain, how to setup the debugger appropriately and hook the application to debug. I will cover more about the process datastructures in my next blog.