Debug symbols, dotPeek and long paths in Visual Studio

In my previous post, I explained how to use the symbol server in dotPeek 1.2 to debug any assembly in Visual Studio, allowing you to set breakpoints and step into any method (provided it was decompiled by dotPeek).

While this is great, I noticed that there was one particular method I couldn’t step into – the moment I tried I got the sadly familiar Source Not Found page:

Clicking the Browse and find… link did nothing, and the Source search information dropdown appeared below. Expanding it, I could see where Visual Studio attempted to load the source file from:

Locating source for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'. (No checksum.)
The file 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs' does not exist.
Looking in script documents for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'...
Looking in the projects for 'C:\Users\Igal\AppData\Local\JetBrains\dotPeek\v1.2\SymbolCache\CSharp\Microsoft.VisualStudio.ProjectSystem.VS.Implementation.pdb\538887009A094E419882756878C69B2A1\Microsoft.VisualStudio.ProjectSystem.VS.Implementation\VisualStudio\ProjectSystem\VS\Implementation\Package\Automation\OAProjectItems.cs'.
The file was not found in a project.
...

And so on. Quick search for OAProjectItems.cs using my most favorite tool, Everything, revealed that it was indeed present in that location, so why couldn’t Visual Studio open it? I decided to open the file manually by pasting its full path into the Start – Run dialog (Win-R), but then I got the following error:

Finally, I tried to go to the file location using the cmd, and I got my answer – the path was simply too long for Windows (and therefore, Visual Studio) to handle!

Windows has an unfortunate MAX_PATH limitation at 260 characters is the source of great pain, and I hope Microsoft fixes it one day. Meanwhile, here’s how to “work around” this particular issue: dotPeek generates debug symbols under %LOCALAPPDATA%\JetBrains\dotPeek\v1.2\SymbolCache\CSharp for C# code, and unfortunately, it isn’t possible to configure. Luckily, it is possible to configure where Visual Studio looks for debug symbols. This is a little-known page in the Solution properties called Debug Source Files:

What I did was copy the entire contents of dotPeek’s SymbolCache\CSharp directory into a local directory d:\sym, and added it to the search list (pictured above). Also, I made sure to delete everything from the bottom list (Do not look for these source files) – if Visual Studio is unable, for any reason, to open a source file, it will add it to this blacklist. It was filled with the files I needed, so I removed them from the list.

After doing this, I could go back to debugging, was was able to step into methods that were previously unavailable!

As the comments below mention, this is also possible to do without copying, by creating a (symbolic) link between the directories! From an elevator command shell, run:

mklink /D d:\sym %localappdata%\JetBrains\dotPeek\v1.2\SymbolCache\CSharp

Thanks guys!

Happy debugging!

How to debug anything with Visual Studio and JetBrains dotPeek v1.2!

Sometimes, we wish we could just step into some 3rd party library, to figure out how it works, but we either don’t have the source code, or otherwise just can’t. Fortunately, this is made possible by dotPeek v1.2 that was just released, which can act as a symbol server for decompiled assemblies!

So let’s suppose we want to put a breakpoint inside Console.WriteLine (or any other method in any other assembly). Here’s what we need to do:

  • Open dotPeek, add the required assemblies to the Assembly Explorer, and press the Start Symbol Server button.

You can configure the port and the symbol generation settings in Tools – Options. The default address is http://localhost:33417/

  • In Visual Studio, go to Tools – Options, then navigate to Debugging – Symbols. Add the location of the dotPeek symbol server.

In addition, make sure that Just My Code (in General) is unchecked, and press OK. Some symbols will be loaded, this might take a few moments.

  • Next, we need to set a breakpoint inside the method which we’re interested in. This can be done with a little-known Visual Studio trick, allowing you to create a breakpoint at any function. Go to Debug – New Breakpoint – Break at Function, and in the dialog enter the fully qualified method name, e.g. System.Console.WriteLine.

After pressing OK, you’ll get a message saying “IntelliSense could not find the specified location. Do you still want to set the breakpoint?”. It’s fine – press Yes.

  • Finally, start your application with the debugger (F5) and you will stop at the breakpoint! You can use all familiar debugging options, such as stepping over/into, watch, autos and the datatip.

Happy Debugging!

How I Improved Sharing My Blog Posts On Twitter With This One Simple Trick!

(I’m so sorry for writing this title! They made me do it, I swear! I won’t ever do it again!)

I came across a wonderful blog post the other day, titled Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison, written by Kristof Kovacs (@kkovacs). Being the twitter addict I am, I immediately felt the need to share this post, and I was met with the eternal twitter dilemma – which part of the title/credit I omit, due to the 140 character limit. I ended up doing this:

Which is what most people would do (chop enough of the title to fit, use a URL shortener for the link, the twitter way). Which didn’t leave room to give credit to the original author of the post (a fact pointed to me by @omervk, thank you very much!)

I was wondering if there was a way to encode this information (author’s twitter handle) within the post itself, using html meta tags. Then I realized that most online tech blogs have this feature available to them – any time you paste a link from, say, Engadget or The Verge on twitter, a preview of the article is visible in the twitter stream.

Turns out, this is available not just for tech blogs, but to everyone! Enter Twitter Cards.

Twitter Cards is exactly what it sounds like – you can embed media and other information to your own tweets! There are many types of cards to choose from, but for my purposes I wanted the Summary card – a simple card including a title, description, thumbnail and the twitter account information. This requires adding various html meta tags to your page, all detailed in the URL above.

To enable Twitter Cards, you will need to Apply for the service, after adding the meta tags on your blog/site. The application process supposedly takes up to 72 hours, but in my case I was approved within minutes! Your mileage may vary.

Here is what I did to enable this for my WordPress blog:
  1. Install Twitter Cards Meta plugin
  2. After installing and activating, go to Settings – Twitter Cards Meta
  3. Fill in your information (e.g. your twitter username)
  4. Go to the Twitter Cards Validation page, and log in with your twitter account
  5. Select desired card type (I selected Summary)
  6. Click the Validate & Apply tab, and paste a link to any one of your blog posts
  7. If you’re satisfied with the results, press Go and after seeing the preview, apply for approval

It might take up to 72 hours to get approved, but usually this happens within minutes (as it was in my case). You will get an email from twitter when you’re approved:

And voilà! Now, every time someone shares a link to a blog post I wrote, it will contain a preview, and my twitter handle!

Turning on Twitter Cards will also give you much better Twitter Analytics – with much more detailed information about clicks and retweets of your cards!

Migrating from TFS to git to Visual Studio Online–the survival guide

This is a step-by-step recount of my attempt to migrate an existing 3-year old TFS (TFVC) repository to git, while keeping all the history, and then moving it to Visual Studio Online, a TFS in the cloud. This wasn’t an easy task, as there is no direct export-import built into either tools. I also ran into some problems during this lengthy process, and I describe the steps to solve them. Set aside a few hours of your time, brew some coffee (or tea), and let’s get started!

Step 1: TFS to git

We first need to export the entire TFS repository to git. This is achieved by “cloning” the entire TFS repository with git-tfs or git-TF, both open source tools. While designed to do the same, mainly, providing a bridge between git and TFS, the former is an older, more mature project, and the latter is a tool created by Microsoft for the same purpose. I initially tried to use Microsoft’s git-TF, but after more than 24 hours of waiting for the clone to end, it died with a Java exception (the tool by Microsoft is written in Java).

First, install git-tfs. It’s best installed with Chocolatey by using cinst gittfs. Next, we need to get the exact name of the project we want to clone. Assuming your TFS server is https://tfs.contoso.com, first, run the following command to list all the branches:

git tfs list-remote-branches https://tfs.contoso.com/tfs/DefaultCollection

This will output all the branches that exist in the DefaultCollection. Branches marked with [*] are the root branches, and these are what we want to clone. Run the following command to clone the required branch, e.g.:

git tfs clone https://tfs.contoso.com/tfs/DefaultCollection $/Project/MyApp/Dev [<directory>]

This will clone the Dev branch (in this case) to a new directory specified in [<directory>]. If [<directory>] is not specified, a new directory called Dev will be created.

The clone operation might take up to several hours, depending on the size of your TFS repository. The cloning operation with git-tfs will pull each individual check-in, and will apply it as a series of git operations, recreating the history exactly as it happened! Git-tfs will also run a git-gc operation every 100 commits, so the new repository size will be kept in check, and obsolete files will be removed.

After the clone operation is complete, you will have a git representation of the TFS repository on your local disk.

Step 2: git to Visual Studio Online

After setting up a free account in Visual Studio Online (VSO), we first need to create a new project, and make sure Git is selected as the source control. After the project had been created, we can go to the CODE tab, and be presented with instructions to either clone an empty repository, or push an existing one. We want to use the second option, so first, set the remote URL in our cloned git repository to point to VSO.

In a command prompt (or use any one of the 3rd-party visual Git clients), go to the cloned git directory and paste the first command, e.g.:

git remote add origin https://contoso.visualstudio.com/DefaultCollection/_git/MyProj

This will set the origin (the remote repository address) to point to our git repository in VSO.

The most important step is next: unfortunately VSO has a timeout limit of 1 hour for any connection, meaning that an attempt to push the entire repository might fail, if the repository is too big! What’s worse, pushing is an atomic operation, it cannot be resumed in case of a timeout, you will have to start from the beginning.

Luckily, there’s a workaround I was managed to find at the bottom of a similar issue in MSDN forums. Basically, we can push our repository in smaller chunks, thus minimizing the chance of a timeout. This is not the ultimate solution, but it worked for me. Our problem is currently being able to push, subsequent cloning should be much faster, since download bandwidth is almost always faster than the upload.

Disclaimer: This is a somewhat advanced git usage, and I must admit, I don’t quite understand it myself fully. Below is a series of commands I ran, based on the mentioned issue, and it worked in my case. YMMV. If you can explain this, please leave a comment below!

First, the issue tells us, we need to get an idea of the number of commits we have, by running the following:

git rev-list --all --count

In my case, the number was 4012, which I assume is the total number of commits.

Next, however, the issue says to run this:

git rev-list master --first-parent --count

Which is the “depth of the ‘first-parent’ lineage of master”, according to the post author. Not quite sure what this meant (the documentation wasn’t very helpful, either), running this command produced a number which was about half the previous one: 2512.

Not knowing exactly what the numbers meant, I decided to try and follow the post’s advice, and split that number into 5 sections, of 500 commits each, and I pushed them in the following order:

git push origin master~2500:refs/heads/master
git push origin master~2000:refs/heads/master
git push origin master~1500:refs/heads/master
git push origin master~1000:refs/heads/master
git push origin master~500:refs/heads/master

And finally:

git push -u origin master

To push the remaining commits, and create a remote tracking branch for master. To make extra sure, I ran this last command again, and got “Everything up-to date” message.

Refreshing the CODE tab in VSO revealed that all my source code was uploaded successfully! I cloned the repository locally, and ran a diff between the two directories in Beyond Compare, just to make absolutely certain that everything was copied properly. It was!

Step 3: VSO to FogBugz

Few more things left to migrate, and those are all the issues on FogBugz to VSO. In the meantime, though, we need to keep our source control integration with FogBugz. Turns out this is also not a simple task. But, this is a blog post for another day. Stay tuned!

When good permissions gone bad–a case of a failed build

I was called over to see if I could help solve a strange issue – every time the build script (Ant) for the client’s Android app ran – certain files that were modified by the build script (a .properties file, few others), were suddenly inaccessible to other people logging to the machine – only the user who initiated the build could still write to the files. Looking at the file permissions tab proved as much: only the current user and the Administrators group could access the file!

My initial investigation into Ant’s build.xml led me to an interesting discovery – all the files that lost their permissions were modified using the ReplaceRegExp task for Ant – a task that could replace text in a file using regular expressions. Quick Google search for “replaceregexp ant file permissions” led me to this similar issue, which was, unfortunately, closed as wontfix.

imageI decided to investigate for myself. Using my go-to tool for this task, Process Monitor, I decided to trace all activity of java.exe (which runs the Ant tasks), looking for anything that has to do with setting permissions or writing the file, filtering just the file I was interested in (project.properties). After running the specific Ant task that did a simple regex replace in the file, it was indeed changed, and its permissions were changed as well.

However, looking at the File Summary window of Process Monitor (Tools – File Summary), I saw that not only there were no new ACL permissions set (Set ACL was called 0 times), no actual bytes were written!

So how was it possible that the file was modified, but nothing was written? Looking back at the events in Process Monitor I also could not see any calls to WriteFile.

Feeling confused, I then started looking for anything that could appear relevant, until one entry in particular caught my eye: a call to SetDispositionInformationFile with a flag Delete: True.

I checked who was calling this API from the Stack tab of this event’s properties, and saw that it was a call to deleteFile, originating in java.dll. This confirmed my suspicion that the file was not written directly at all – but replaced with another file, possibly from a temp directory.

My suspicions proved to be correct – looking at the source code for ReplaceRegExp task I saw that it was exactly how it did it: by using

FILE_UTILS.createTempFile("replace", ".txt", null, true, true)

to create a temporary file, and then later renaming that temporary file with the original file name!

FILE_UTILS turned out to be just a wrapper calling Java’s File.createTempFile. The 3rd argument, which was null in this case, tells createTempFile where to create the file. If null is passed, it will use the default %TEMP% environment variable.

Which ended up explaining the problem exactly – by default, it uses the local user’s %TEMP% directory to create the temp file that replaces the original one. The default per-user %TEMP% directory located in the %LOCALAPPDATA% directory of the user’s profile – meaning only the current user (and local Administrators) can access it! This means, that any file created in this directory will inherit the same permissions! In our case, ReplaceRegExp’s implementation caused the per-user temp file to overwrite the file that was in a public folder, causing it to lose all permissions!

A quick workaround for the problem was to set the TEMP directory to c:\temp, temporarily, during the build.

Next time your Ant script causes issues with file permissions – make sure your files are not replaced under your nose.

How to enable login verifications on Twitter from “unsuppored” countries

It’s funny how a random twitter rant can yield valid solutions. One such rant between @omervk (who co-runs plaintextoffenders.com) and myself regarding enabling Two Factor Authentication (2FA, or “login verification”) on twitter being unavailable for people in Israel, caught the attention of Per Thorsheim, an independent security consultant and founder of the Passwords conference. Per, turns out, was interested in this problem because of another matter, that Twitter seemingly turned off login verification for people who do not have their phone numbers associated with Twitter! Per wrote about his own experiences trying to enable 2FA on twitter here.

So what is the problem exactly? In order to enable login verifications on Twitter, turns out, you need to either associate your mobile phone number with twitter, or enable sending login verifications to your iOS or Android twitter app. The latter, turns out also requires your phone number!

Here’s how it supposed to work: you go to the Security and privacy settings in your twitter account. If no phone number is associated, you will see these options:

Notice that both “Send…” options are disabled. Pressing the add a phone link takes you to a page where you can enter your phone number (after selecting your country and carrier), however in my case, I got this message after submitting:

OK, so I can’t add a phone number on the twitter website. Let’s try enabling login verifications on my (Android) twitter app. Go to Settings, your twitter username, then press Security:

What! I am still being asked to add a phone number!

This is the part where I initially gave up, but it turned out that clicking Add phone takes you to the mobile twitter page (in your phone’s browser), where you can enter your phone number, and in my case twitter accepted it, and sent me a verification SMS!

The UX is confusing at this point, after adding my number, I got an SMS with a 6 digit verification code (and short t.co URL to continue verification), but no place in the page to enter it! In fact, I was asked to remove the phone number at this point:

Very odd. However, pressing the manage link above (in the yellow bar) took me to the actual verification page where I was asked to enter the code from the SMS message:

And after pressing Verify, I got a message confirming that my phone number was successfully added!

At this point, after going back to the Security and privacy settings on twitter, I saw that my phone number was now added, and could turn on login verifications!

Also, going to the Security settings on my Android twitter app were now also working!

In conclusion, the steps above may work for you, even if you can’t add your phone number on the twitter website. Did it work for you? Please leave a comment with your country/carrier, and any additional information!

Good luck, stay secure!

How to remap calc.exe to scriptcs.exe

Lately I found myself launching ScriptCS more and more to do simple calculations. I half-jokingly said on twitter that I’d better remap calc.exe to scriptcs.exe on my machine. However it seems that my joke tweet was taken seriously by some people, and I was asked how this was done. So here goes!

For this next trick I will use my most favorite Windows trick – the Image File Execution Options (IFEO). I’ve blogged about IFEO in the past, it’s generally used to allow attaching a debugger to a process before it starts, but can also do other useful things, such as replacing the Windows Task Manager with Process Explorer, or even disabling some processes launching, which is useful to prevent Narrator in Windows 8 from launching via the Win-Enter key.

So here is how to remap calc.exe to launch ScriptCS instead. First, locate on your machine where scriptcs.exe is installed, as we need the full path. You can use the command where scriptcs.exe in CMD to find it. If you don’t have scriptcs.exe in path, best install it via Chocolatey (you’re welcome!)

  1. Open the registry editor (regedit.exe), and navigate to:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
  2. Create a new key called calc.exe
  3. Inside the newly created key, create a new String Value called Debugger
  4. Double-click Debugger, and set c:\path\to\scriptcs.exe -repl as the value

That’s it, from now on when you launch calc.exe, scriptcs.exe will open instead! To undo this, simply delete the key calc.exe from the registry path above.

P.S. if you have Windows SDK installed, you can use the utility gflags.exe to do this instead:

  1. Launch gflags.exe (need to be launched elevated)
  2. Go to the Image File tab
  3. In the Image text box, write calc.exe and press TAB (I know)
  4. Down at the bottom, under Debugger, write c:\path\to\scriptcs.exe -repl and press OK

Bonus: now replace devenv.exe with scriptcs.exe!

Happy hacking!

Code, music and single malt whisky!