The creation of a museum for North Carolina Central University had been a goal early in the college's history. James E. Shepard, then the president of the institution, understood the importance of the fine arts in building cultural pride by preserving and collecting works of art. Although the more pressing matters of the college took precedent over the museum's creation in the 1920s and 1930s, the steps toward establishing the museum began to take shape in 1942 when Marion C. Parham was hired as an instructor for the art department. Parham is credited with mounting the first successful student exhibition in 1954 in room 206 of the Music and Fine Arts Building, which is located adjacent to the museum's current location. The next art department chair, Edward Wilson, was also an aggressive advocate for gallery space. In 1958 an exhibition space was set aside in one room of the Fine Arts Building and continued to be used until 1964.

In 1981 Lynn Igoe became the director of the art gallery and in 1972 the first gallery was formed using renovated space from the old cafeteria. Nancy C. Gillespie, Igoe's successor pressed on for a permanent exhibition space and outlined a design which would allow space for temporary exhibitions and the emerging permanent collection. However, in 1976 under the leadership of Norman Pendergraft, the next art museum director, the museum's collection grew to more than 200 works of art. The collection included works by both emerging and established African-American artists as well as works by other artists documenting the African-American experience.

On November 4, 1977, the university broke ground for a new art museum in its current location. In 1986, Irwin Belk made a donation of $59,500 to establish an endowment for restoration and acquisitions costs. For this reason the museum's permanent collection has been named the Carol Grotnes Belk Gallery. The collection includes works by noted artists such as Henry O. Tanner, Robert S. Duncanson and Edward M. Bannister. Among its early twentieth century masters are such names as Richmond Barthe, Robert Blackburn, Jacob Lawrence, William H. Johnson, Aaron Douglas, Romare Bearden, Selma Burke, Norman Lewis, Elizabeth Catlett and Charles White. Contemporary artists include Juan Logan, Barkley Hendricks, Minnie Evans, Sam Gilliam and Kerry James Marshall.