From Policy Concepts in 1000 Words: Institutions and New Institutionalism:
[see also Policy in 500 Words: Feminist Institutionalism]
[see also Policy concepts in 1000 words: Institutional memory (guest post)]
The study of public policy would be incomplete without an understanding of policymaking institutions. The study of political science would also be incomplete without turning our understanding of terms such as ‘institutions’ upside down. ‘Institution’ may in the past have referred to organizations such as legislatures, courts and executives. With ‘new institutionalism’, it refers to two factors: regular patterns of behaviour; and the rules, norms, practices and relationships that influence such behaviour.
These rules can be formal, or enshrined in a constitution, legislation or regulations:

The constitutional nature of political systems – such as confederal or federal; federal or unitary; presidential, parliamentary or semi-presidential; unicameral or bicameral; containing constitutional courts; or holding procedures for regular referendums.
Their operating procedures – including electoral systems, party systems, rules of government formation and executive–legislative relations, the role of public bureaucracies, and the extent to which group-government relations are ‘institutionalised’ (such as in formal corporatist arrangements).
Their regulatory frameworks – including the rules governing the operation of economic organizations, interest groups, and public organizations, and the rules governing the provision of public services.

Rules can also be informal, and are described variously as habits, norms, practices or rules that develop without a grand plan. As such, they are often unwritten and difficult to identify or understand by people outside of an organisation.
In practice, we may identify a mix of formality and informality – the combination of written regulations and unwritten understandings of how organisations are expected to operate. This helps explain why political systems often operate rather similarly in practice despite having different constitutional arrangements. For example, the commonly perceived logic or benefit of subsystem/ policy community arrangements helps explain why they are central to most systems.
So far, so good. The problems begin when we try to move from this rather intuitive and broad discussion, to produce concrete studies and detailed approaches. There are three main problems to look out for:
. We may not know what an institution is. Instead, we often use the ‘I know it when I see it’ approach. For example, the Oxford Handbook of Political Institutions dedicates at least one chapter to: the state, civil society, economic institutions, constitutions, federal and territorial institutions, executives, legislatures, courts, bicameral structures, public bureaucracies, the welfare state, regulations, local government, political parties, electoral systems, direct democracy, international and non-governmental institutions. This is a wide range of activity, brought together largely because definition of institution is vague.
2. We may not agree what new institutionalism is. 
[see the post for more]