This may be old news for some of you out there but for those of you who are just starting to use the AjaxControlToolkit I wanted to show how you can nest the ModalPopup inside a GridView.  This technique has worked for the most part.  However, I will caution if you show a lot of records within the gridview  it could slow up performance on rendering the page.  Keep in mind this technique below is adding the ModalPopup to each row within the gridview. 

First you need the following style within the page your CSS sheet

.modalBackground
{
    background-color:#f5f5f5;
    filter:alpha(opacity=80);
    opacity:0.7;
}

Second you will notice we have the AjaxControlToolkit registered on the aspx page, along with a ScriptManager

<%@ Register Assembly="AjaxControlToolkit" Namespace="AjaxControlToolkit" TagPrefix="ajaxToolkit" %>

<asp:ScriptManager ID="ScriptManager1" runat="server">
</asp:ScriptManager>

Third, you will see the GridView with one TemplateField that has a LinkButton, Panel, Button and the ModalPopupExtender.

The LinkButton is set as the TargetControlID
The Panel is set as the PopupControlID
The Button is set as the OkControlID

<asp:GridView ID="gvSchedule" runat="server" AutoGenerateColumns="false" ShowHeader="false" GridLines="None" HorizontalAlign="Center">
        <Columns>
            <asp:TemplateField>
                <ItemTemplate>
                    <table cellpadding="5" cellspacing="0">
                        <tr>
                            <td>
                                <asp:LinkButton ID="lnkBtnViewDetails" runat="server" Text="View" />
                                <asp:Panel runat="server" ID="pnlDetails" style="display: none; width: 500px; border: solid 1px black; overflow: scroll; height: 400px; background-color: White;">
                                     <asp:Literal ID="Literal1" runat="server" Text='<%# (string)Eval("strDetails") %>'></asp:Literal>                                           
                                </asp:Panel>
                                <ajaxToolkit:ModalPopupExtender ID="modalPopUp1" runat="server"
                                    TargetControlID="lnkBtnViewDetails"
                                    PopupControlID="pnlDetails"
                                    BackgroundCssClass="modalBackground"
                                    DropShadow="false"
                                    OkControlID="btnCloseDetails" />
                            </td>
                        </tr>
   <tr>
                            <td style="text-align: center;">
                                 <asp:Button ID="btnCloseDetails" runat="server" Text="Close Details" />
                            </td>
                        </tr>
                    </table>
                </ItemTemplate>
            </asp:TemplateField>
        </Columns>
        <AlternatingRowStyle CssClass="gridAltRowStyle" />
 </asp:GridView>

This is pretty much all you need to utilize the ModalPopup nested within a GridView. 

Note: I handled the GridView binding in the code behind, or you can handle it with a SqlDataSource.

Hope this has helped on how you can utilize the ModalPopup inside a gridview.