PTVS has been a long time in the making and it represents the fruition of a lot of effort here at Microsoft to produce a Python IDE.  This actually goes back a long time starting it's development several years ago on the IronPython team.  Back then we had done several small projects to figure out what we'd want to do in the Python IDE space.  That included a couple of attempts of building a stand alone IDE using the same components Visual Studio is built upon as a few different attempts at extending Visual Studio to add Python support (some of this having seen the light of day in the form of IronPython Studio and the Python integration which ships w/ the VS SDK).  Ultimately we were able to re-use bits and pieces from all of these attempts and release IronPython Tools for Visual Studio w/ the Alpha of IronPython 2.7. 

But we needed one last push to turn PTVS into what you see today - and that final push brought support for more than just IronPython and we now have turned Visual Studio into a general purpose Python IDE.  Whatever version of Python you'd like to use I think you'll find that PTVS provides a great experience - whether you're using traditional CPython or IronPython (which we still have special support for including WPF designer support) or another Python distribution such as the speedy PyPy.  The only feature which doesn't currently work across Python distributions is the profiling support which for performance reasons is tied to the CPython embedding API.

Anyway, if you're looking for doing Python development on Windows I hope you'll give PTVS a shot and let us know what you think.

Here's the full release notes if you'd like more details about what's changed:

 Hello Python Community,

We’re proud to announce the release of IronPython 2.6 final.  This is a major release with improvements across all areas of IronPython and can be downloaded from https://ironpython.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=12482.  Significant changes include:

·         Updating the language and standard library to match CPython 2.6

·         Improved .NET integration

·         Updating to the latest version of the DLR

·         Adding previously missing CPython features and fixing bugs

·         Performance improvements in particular startup time improvements

Python 2.6 support includes a large number of new features which include support for the new bytes and byte array types (PEP 3112), decorators for classes (PEP 3129), advanced string formatting (PEP 3101) which will feel familiar to .NET users and integrates nicely with IFormattable, print as a function (PEP 3105), Abstract Base Classes (PEP 3119), support for binary literals, along with lots of other minor features and changes.

IronPython also continues to improve .NET integration in this release.  We’ve added the __clrtype__ method to types which enables deep integration via meta-classes with the .NET type system.  There are also a number of smaller changes such as supporting IDisposable in for loops matching the behavior of other .NET languages.  This release also includes the latest version of the DLR and fixes a number of issues related to cross-language dynamic interop.  Not to be left out there’s improvements in Silverlight integration by supporting Python code in HTML script tags.

We’ve also continued to improve Python compatibility by adding missing features and fixing bugs.  In this release we’ve added support for the ctypes module which provides interop with native code in a compatible way across Python implementations.  We’ve also implemented sys.settrace to provide support for the pdb module and other Python debuggers.  This release also changes how we support sys._getframe: a fully working version is now available by a command line option;  when not enabled sys._getframe doesn’t exist at all. This release also fixes over 400 bugs removing a large number of smaller incompatibilities.

As always we’ve also continued to improve performance, and this release primarily focuses on improving startup time.  The IronPython installer now pre-compiles (ngens) IronPython during installation on both 32-bit and 64-bit platforms.  Modules are now interpreted initially and hot functions are compiled for faster import times.  A number of other code paths that used to involve runtime code generation have been optimized to be contained within the pre-compiled IronPython binaries.  We’ve also made a number of smaller changes which improve performance in other areas such as adding constant folding.

Finally we’d like to thank everyone who reported issues and helped make this a better release:  Anders M. Mikkelsen, Dan Eloff, Zachc, yamakox, vernondcole, VAks, tscottw, tonyandrewmeyer, tomwright, TomasMatousek, tkamiya, timers, srivatsn, sopeajw, saveenr, sanxiyn, rridge, ronniemaor, quirogaco, pythonfoo, pysunil, pm100, pl6306, paulfelix, orestis, olegt, oldman, NDHUMuscle, mycall, mmaly, mmacdonaldssfcu, maplpro, luntain, llaske, lbaker, Lawouach, laurionb, laughingboy, kurhan, kuno, kowenswp, klrohe, kevgu, jmesserly, jlunder, jdhardy, jbevain, jackeyoo, hhonisch, gz, gjones, fwereade, deadalusai, daveremy, Seo Sanghyeon, CurtHagenlocher, chaghi, cgravill, cartman, bobarnso, atifaziz, ashcor, alvanet, Helmut, fuzzyman, fabiofz, Eloff, RuiDC, Kevin Chu, Kyle Howland-Rose egonw, davec, dungen, dsblank, cjacobs, dmajnemer, leppie, Mark Rees, soulfry, tatwright, ufechner and wilberforce.

Thank you for your continued support of IronPython.

The IronPython Team

 https://ironpython.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=28125#DownloadId=93331

 This is built using Visual Studio 2010 Beta 2 and the IronPython CTP for that release - so you'll need to download both of those.  I've also included the Python and Ruby code that I used during the demonstration in a file called code.txt.  The paint program it's self is in the final form at the end of the demo.

 First let's go ahead and create our LanguageContext:

public class MyContext : LanguageContext {
    public MyBinder Binder;
    public MyContext(ScriptDomainManager manager, IDictionary<string, object> options) : base(manager) {
        Binder = new MyBinder(manager);
    }
   
    public override ScriptCode CompileSourceCode(Microsoft.Scripting.SourceUnit source, Microsoft.Scripting.CompilerOptions options,Microsoft.Scripting.ErrorSink errorSink) {
        throw new NotImplementedException();
    }
}

For now you're going to have to figure out how to parse and compile code on your own.  But the interesting point I'd like to highlight right now is the MyBinder class which is implemented like this: 

public class MyBinder : DefaultBinder {
    public MyBinder(ScriptDomainManager manager) : base(manager) {
    }
    public override bool CanConvertFrom(Type fromType, Type toType, bool toNotNullable, NarrowingLevel level) {
        return toType.IsAssignableFrom(fromType);
    }
   
    public override Candidate PreferConvert(Type t1, Type t2) {
        return Candidate.Ambiguous;
    }
}

 With this class implemented I can now start doing .NET binding when I'm trying to resolve method calls, member accesses, or other .NET APIs.  But if I'm going to be calling methods (and you'll frequently call methods if you're resolving members - because methods are methods, properties are methods, events are implemented using methods, etc...) then you'll need something which also chooses which overload of a method to call.  For this we have a overload resolver factory and and overload resolver.  You need to give the binder the factory and it'll then create a resolver for a set of methods which need to be resolved:

internal sealed class DefaultOverloadResolverFactory : OverloadResolverFactory {
    private readonly DefaultBinder _binder;

    public DefaultOverloadResolverFactory(DefaultBinder binder) {
        _binder = binder;
    }

    public override DefaultOverloadResolver CreateOverloadResolver(IList<DynamicMetaObject> args, CallSignature signature, CallTypes callType) {
        return new DefaultOverloadResolver(_binder, args, signature, callType);
    }
}

Ok, now we've got a language, a binder, and an overload resolver.  So at this point we're ready to implement a normal DLR binder.  There's a bunch of base classes which can be derived from for the various protocols which are supported by the DLR.   Here I'm going to implement a get member and invoke member binder to give a quick demonstration of how you interact with a DefaultBinder subclass. 

public class MyGetMemberBinder : GetMemberBinder {
    private readonly MyBinder _binder;
    public MyGetMemberBinder(MyBinder binder, string name) : base(name, false) {
        _binder = binder;
    }
   
    public override DynamicMetaObject FallbackGetMember(DynamicMetaObject target, DynamicMetaObject errorSuggestion) {
        var resolver = new DefaultOverloadResolverFactory(_binder);
        var res = _binder.GetMember(Name, target, resolver);
        if (res.Expression.Type.IsValueType) {
            res = new DynamicMetaObject(
                Expression.Convert(res.Expression, typeof(object)),
                res.Restrictions
            );
        }
        return res;
    }
}

public class MyInvokeBinder : InvokeBinder {
    private readonly MyBinder _binder;
   
    public MyInvokeBinder(MyBinder binder, CallInfo callInfo) : base(callInfo) {
        _binder = binder;
    }
   
    public override DynamicMetaObject FallbackInvoke(DynamicMetaObject target, DynamicMetaObject[] args, DynamicMetaObject errorSuggestion) {
        var resolver = new DefaultOverloadResolverFactory(_binder);
        return _binder.Call(new CallSignature(CallInfo.ArgumentCount), resolver, target, args);
    }
}

Finally we're ready to put it all together.  We can create a ScriptRuntime which has our language setup, we can get our language context, and then create a few CallSites which will use our newly created binders:

public class Test {
    public static void Main() {
       ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
       setup.LanguageSetups.Add(CreateLanguageSetup());
       var sr = new ScriptRuntime(setup);
       var engine = sr.GetEngine("test");
       var myCtx = (MyContext)HostingHelpers.GetLanguageContext(engine);
       var binder = myCtx.Binder;
      
       CallSite<Func<CallSite, object, object>> getMemSite = CallSite<Func<CallSite, object, object>>.Create(new MyGetMemberBinder(binder, "Length"));
       Console.WriteLine(getMemSite.Target(getMemSite, new[] { 1, 2, 3}));
      
       CallSite<Func<CallSite, object, object, object, object>> invokeSite = CallSite<Func<CallSite, object, object, object, object>>.Create(new MyInvokeBinder(binder, new CallInfo(2)));
       Console.WriteLine(invokeSite.Target(invokeSite, new ParamsDelegate(ParamsMethod), 1, 2));
      
    }
   
    public delegate object ParamsDelegate(params object[] args);
    public static object ParamsMethod(params object[] args) {
        foreach(object o in args) {
            Console.WriteLine(o);
        }
        return 42;
    }
   
    public static LanguageSetup/*!*/ CreateLanguageSetup() {
            var setup = new LanguageSetup(
                typeof(MyContext).AssemblyQualifiedName,
                "test",
                new[] { "test"},
                new[] { ".tst" }
            );

            return setup;
        }
}
 

Viola, we've created the start of a language as far as the DLR is concerned. We don't yet support parsing and compiling code but we can interact with .NET objects.  There's still a bunch of things we could customize on both the DefaultBinder and OverloadResolver to customize how the binding and calls occur.  But that'll have to wait until a future blog.

The first release was IronPython 2.0.2 which just includes bug fixes for issues which have been particularly irritating to our users.  This is a very minor release and doesn't include breaking changes.  Probably the most significant fix is that we've fixed ngen on 64-bit platforms.  Previously when you would install 2.0 and enable ngen it would only ngen the binaries for 32-bit assemblies.  Now we'll ngen for 64-bit which should help users running on 64-bit see much faster startup times.  I hope that this will make 2.0 much more usable until we get the really big improvements that 2.6 brings.

And speaking of 2.6 the other release this week was our 2nd and final beta.  2.6 is shaping up to be an exciting release.  We've significantly improved the startup time of IronPython - this has been our number one requested change.  We've implemented more standard modules and other missing functionality to be closer to CPython.  This includes the new ctypes module, support for sys.settrace and pdb, as well other new functionality that 2.6 has introduced such as the underlying support for the io module.  And finally we've added a bunch of new features to improve .NET interop as well - we've now got support for generic method type inference and added the core support that will enable using .NET attributes and in general working better w/ .NET when static types are expected.  I'm pretty excited about this release and can't wait to get the final version out - so please try out this beta and let us know if you run into any issues.

But before I look at what’s going on here I’d like to briefly say we love to hear feedback from users of IronPython.  We’ve have a mailing list you can join and give us your feedback on and bugs can be reported on our website.  We track both of these very closely and respond as quickly as we can – which is usually pretty fast.  I just mention that because it’s easier for us to improve IronPython when people let us know about a problem vs. us having to scour the web to find issues.  Hopefully this piqued interest will have more people trying out IronPython and giving us feedback.

Ok, but let’s get onto the fun technical details!  It all comes down to one line of code “Node.counter += 1”.  That’s a pretty innocent looking piece of code, how can it cause so many problems?  First you need to understand a little bit about how IronPython works.  IronPython internally keeps versions of type objects and uses this information to generate optimal code paths for accessing those objects.  So the next line of code in this example, “self._children = children” usually turns into something that’s like:

What that one line of code causes then is for the version of the type to change and for this “rule” to be invalidated.  So we’ll figure out what to do, generate a new cached rule, only to have it not work the next time around.  The same thing applies to the calls to Node – we re-generate the “optimal” code again and again only to discover we have to throw it away.

So that sounds pretty bad but if you’re familiar with Python’s member access rules you’ll quickly recognize that this will be much faster than needing to do the resolution again and again.  That involves looking through the class hierarchy to see if we have any data descriptors or classes defining __setattr__ and it's nice to avoid all of that work.   So to get the best performance we require for the world to settle down and not be continuously changing.  We’re well aware of this problem with our implementation and we have continued to focus on improving it.  IronPython 2.0 already includes one mitigation so that we don’t need to continuously generate new code – instead we effectively patch new values into an existing piece of code.  In our IronPython 2.6 branch we’ve continued to improve performance here and significantly reduced the expense when things get mutated.  But it turns out that there’s an easy fix for this particular problem we’ve just never thought it that important of a problem to solve.

One of the amusing things about this is we’ve actually seen it before – Resolver Systems was having some performance issues with Resolver One  which is their advanced spreadsheet which is written in IronPython.  One of those issues turned to be a type which they both used frequently and mutated frequently.  This actually doesn’t seem to be a common idiom in Python code which is the reason we’ve never optimized for it before.  In this case though it turned out that they had found an optimization that improved performance on IronPython 1.0 but was destroying perf on IronPython 2.0!  For Resolver fixing this was a simple change for them to no longer mutate types.  Even if you really want to use a type as a property bag you can always move that to a type which you aren’t creating instances of and executing methods on.

With all that in mind let’s look at some numbers.  Here I’ve compared IronPython 2.01, IronPython 2.6 Beta 1, CPython 2.6 plus IronPython 2.6 with a fix for this issue.

Like Robert’s charts I have tree size in number of nodes across the bottom and execution time in seconds across the top and the execution times are logarithmic.  We can see the abysmal performance of 2.0.1, we can see that 2.6B1 has already made significant improvements, and with IronPython 2.6B1 plus a fix for this specific issue we actually beat CPython as the number of iterations scales up.

Here you can see that all 3 versions of IronPython are performing about the same and are coming in significantly faster than CPython as the number of iterations goes up.  These are now also matching the numbers Robert is getting when he has made the same change.

So you might be curious - what’s the fix?  Well in this case we’re just mutating the type by changing a simple value stored in the type (versus say adding a property or a method to the type).  Today we get a very slight throughput performance benefit because we can burn the value directly into the generated code instead of fetching it from the wrapper the type object keeps it in.  With the fix we’ll burn the wrapper in and fetch the value from the wrapper each time.  Then when we go and modify the type we just modify the wrapper instead of the type and its version is unchanged!  This fix is now checked into the IronPython source tree and you can download the fixed sources right now. 

In conclusion I’d like to wrap up a little bit about our performance philosophy – performance is hard and deciding which numbers matter to you is important.  We have tried to tune IronPython for apps which run for more than a fraction of a second.  We have also tuned IronPython for code which is not constantly mutating types at runtime.  And we’re always open to feedback when users encounter problems with IronPython and we’ll listen to what scenarios are important for them.  But if you look at these graphs I think you can see that we’ve done a good job at making IronPython a fast implementation of Python.

There’s a few key pieces you need to understand here.  First you’re returning an expression tree back to the DLR to tell the DLR what to do.  The DLR will compile this expression tree and run it.  You’re also returning a set of “restrictions”.  These will be evaluated by the DLR to see if it can re-use the same code for other objects in the future.  You’re unlimited in what you can do with restrictions as they can be arbitrary Expression trees.  But here I’m simply restricting based upon the arguments .NET types which is one of the most common restrictions.  Both the expression tree and restrictions get packaged up in a DynamicMetaObject which is conveniently the same type of object you receive as your arguments. 

Finally there’s the hash identity which is how the DLR knows if it can share rules or not amongst different binders.  The base DLR binders override GetHashCode/Equals to hash on the operation, member name, etc… depending on the binder type so I just return this here.

So here it is a simple binder.  This targets the latest and greatest DLR sources which has experienced some renames and so isn’t compatible with IronPython 2.0.  But you can always get the latest IronPython source code which this will compile against at https://www.codeplex.com/IronPython/SourceControl/ListDownloadableCommits.aspx

    class MyOperationBinder : BinaryOperationBinder {

        public override DynamicMetaObject FallbackBinaryOperation(DynamicMetaObject target, DynamicMetaObject arg, DynamicMetaObject errorSuggestion) {

            if (target.LimitType == typeof(int) && arg.LimitType == typeof(int)) {

                return new DynamicMetaObject(

                    Expression.Add(

                        Expression.Convert(target.Expression, typeof(int)),

                        Expression.Convert(arg.Expression, typeof(int))

                    ),

                    BindingRestrictions.GetTypeRestriction(target.Expression, typeof(int)).Merge(BindingRestrictions.GetTypeRestriction(arg.Expression, typeof(int)))

                );

            }

            return new DynamicMetaObject(

                Expression.Throw(

                    Expression.New(

                        typeof(Exception).GetConstructor(new Type[] { typeof(string) }),

                        Expression.Constant(

                            String.Format("Can't perform operation {0} on {1} and {2}", this.Operation, target.LimitType, arg.LimitType)

                        )

                    )

                ),

                BindingRestrictions.GetTypeRestriction(target.Expression, typeof(int)).Merge(BindingRestrictions.GetTypeRestriction(arg.Expression, typeof(int)))

            );

        }

        public override object CacheIdentity {

            get { return this; }

        }

    }

 Basically this just uses the .NET System.Data namespace to do the communication with the database server.

import clr
clr.AddReference('System.Data')
from System import Data
from System.Data import SqlClient
import System

DatabaseError = Data.DataException

from threading import local

class SqlCursor(object):
    arraysize = 1
   
    def __init__(self, connection):
        self.connection = connection
        self.transaction = None
        self.reader = None
        self.record_enum = None
   
    def execute(self, sql, params=()):       
        parameters = []
       
        # translate to named parameters
        if type(params) in (list, tuple):
            # indexed params, replace any %s w/ @GeneratedName#
            if sql.find('%s') != -1:
                cmd = ''
                sqlSplit = sql.split('%s')
                for text, value, index in zip(sqlSplit, params, range(len(params))):
                    cmd += text + '@GeneratedName' + str(index)
                    parameters.Add(SqlClient.SqlParameter('@GeneratedName' + str(index), str(value)))
                
                sql = cmd + sqlSplit[-1]
        else:
            for name, value in params.iteritems():
                sql = sql.replace('%(' + name +')s', '@' + name)
                parameters.Add(SqlClient.SqlParameter('@' + name, str(value)))
       
        command = SqlClient.SqlCommand(sql, self.connection)
       
        for param in parameters: command.Parameters.Add(param)
       
        self.record_enum = None
        self.reader = command.ExecuteReader()
   
    def close(self):
        self.reader.Close()
   
    def executemany(self, sql, param_list):
        res = []
        for s in sql:
            res.append(execute(s, param_list))
        return res
       
    def fetchall(self):
        return [self._make_record(record) for record in self.reader]           
   
    def fetchone(self):
        if self.record_enum is None:
            self.record_enum = iter(self.reader)
        if self.record_enum.MoveNext():
            return self._make_record(self.record_enum.Current)
        return None
       
    def fetchmany(self, size=None):
        if size is None: size = SqlCursor.arraysize
        res = []
        for i in range(size):
            x = self.fetchone()
           
            if x is None: break
           
            res.append(x)
        return res

    def _make_record(self, record):
        return tuple((self._fix_one_record(record[i]) for i in xrange(record.FieldCount)))

    def _fix_one_record(self, record):
        if type(record) is System.DateTime:
            return datetime.datetime(record)
       
        return record
   
    @property
    def rowcount(self):
        if self.record is not None:
            return self.reader.RecordsAffected
        return -1
   
class DatabaseWrapper(local):
    def __init__(self, **kwargs):
        self.connection = None
        self.queries = []
        self.transaction = None

    def cursor(self):
        from django.conf import settings
        if self.connection is None:
            if not settings.DATABASE_HOST:
                settings.DATABASE_HOST = "(local)"
            if settings.DATABASE_NAME == '' or settings.DATABASE_USER == '':
                conn_string = "Data Source=%s;Initial Catalog=%s;Integrated Security=SSPI;MultipleActiveResultSets=True" % (settings.DATABASE_HOST, settings.DATABASE_NAME)
            else:
                conn_string = "Data Source=%s;Initial Catalog=%s;User ID=%s;Password=%s;Integrated Security=SSPI;MultipleActiveResultSets=True" % (settings.DATABASE_HOST, settings.DATABASE_NAME, settings.DATABASE_USER, settings.DATABASE_PASSWORD)
           
            self.connection = SqlClient.SqlConnection(conn_string)
            self.connection.Open()
           
        cursor = SqlCursor(self.connection)
        if settings.DEBUG:
            return util.CursorDebugWrapper(cursor, self)
        return cursor

    def _commit(self):
        if self.transaction is not None:
            return self.transaction.Commit()

    def _rollback(self):
        if self.transaction is not None:
            return self.transaction.Rollback()

    def close(self):
        if self.connection is not None:
            self.connection = None
            self.transaction = None

I'll be switching from my current test role over to development to work on on IronPython and improving the CLR's support for scripting languages in general.  I started on the new team last Monday and it's already been a lot of fun.  Today we've shipped our first release since I joined (0.9.5 - which you can download here.  While this is the first release since I've joined the team it's the second release to which I've contributed.  This release brings a bug fixes, support for several new built-in types and improvements when using Python objects from WPF. 

So expect to see less about reliability and hosting (when I post :) ) from me over time and more about IronPython and scripting.  I still have some more I want to say about reliability and hosting so there should be an interesting mix going forward.

Anyway, I encourage everyone to check out our latest release and let us know what you think!

That feature which most readers of my blog are probably now familiar with is fiber mode.  So why did we cut fiber mode for the released version of the product?  And what changes will you see because of this?

We had gone through great efforts to make sure that all of our core functionality worked great inside of a fiber mode host.  Not only did the CLR team do a lot of work to make sure fiber mode was solid, but so did many other teams who picked up our toolset and verified that worked properly by running their tests inside of fiber mode.  Not to mention the teams that typically run their tests inside of Yukon who also make sure their features work in fiber mode.  We even spent extra time at the end of beta 2 to make sure we got all the last minute fiber bugs fixed from our full fiber mode test pass - but ultimately it wasn't enough.

In the end we had one final exit criteria we'd have to pass in order to make sure fiber mode was solid:  Stress.  Between the choices of spending our time fixing the bugs that we knew ALL of our customers would hit in thread mode, or fixing the bugs that a small percentage of users would hit in fiber mode, we picked fixing thread mode bugs.  In the end I think most customers will be happy as the SQL and CLR teams came together and we've spent a huge amount of time, resources and effort ensuring that SQL/CLR works great under thread mode.

For those of you wanting to develop a fiber mode CLR solution I think you need to first ask yourself why you're doing this.  If you're attempting to conserve stack space then this is not the solution you're looking for.  If you're attempting to reduce the number of context switches experienced then you can still get much the same results using thread mode and blocking "switched out" tasks on an event that gets signaled when it's their turn to run.  And of course there's a 3rd, although usually less desirable, option to redesign the way you're approaching the problem of a large number of work items.

Finally what are the changes in removing fiber mode?  Surprisingly everything will work exactly the same as it did in pre-RC and RTM builds except for one API will now return E_NOTIMPL.  That one API is ICLRTask::SwitchOut.  You still can do all the interesting stuff with integrating the CLR task management with your own environment, and you can still switch a logical task in on a thread that previously had another logical task running on it, but you can't switch out a logical task during the middle of its lifetime. 

I realize removing this feature will cause some of you some pain but I hope you'll see that this was the right call for us to get a super stable Whidbey out the door in a reasonable timeframe.